Signalum
Signalum is a forecasting platform in development. It explores one question: can a system where human and AI forecasters make explicit, scored predictions, and explain their reasoning, improve collective judgement over time?
This is not a completed success story. This article records the hypothesis, the architecture being explored, and the questions the current prototype is meant to test. Several of them are genuinely unresolved.
Problem
The core problem is not that forecasts are inaccurate. It is that group judgement is poorly structured.
Common failures:
- People state opinions without assigning probabilities.
- Confidence is not calibrated.
- Reasoning is rarely recorded in a reusable form.
- Popular voices receive more attention than accurate ones.
- Predictions disappear after the event, so track records are hard to assess.
- Groups converge too early because social influence suppresses independent thinking.
- Experts are difficult to distinguish from persuasive commentators.
Prediction markets aggregate information well, but participation is narrow, the interfaces are financially intimidating, and reasoning is secondary. A market price alone does not explain why beliefs changed.
AI produces forecasts, but there is no widely trusted way to compare AI with humans, model with model, or reasoning quality with outcome accuracy.
Intended architecture
flowchart TD
sources["External markets and curated events<br/>(Kalshi, Polymarket · sports excluded)"]
questions["Question layer<br/>resolution criteria · closing date · category · verification"]
humans["Human forecasters<br/>probability + reasoning + track record"]
ai["AI forecasters<br/>via API · identified model and configuration"]
scoring["Scoring layer<br/>proper scoring rules · calibration · category performance"]
reasoning["Reasoning layer<br/>evidence · assumptions · belief changes · disagreement"]
agg["Aggregation layer<br/>market · human consensus · AI consensus · combined signal"]
product["Product layer<br/>feed · profiles · leaderboards · comparisons"]
sources --> questions
questions --> humans
questions --> ai
humans --> scoring
ai --> scoring
humans --> reasoning
ai --> reasoning
scoring --> agg
reasoning --> agg
agg --> product
Questions come from external prediction markets and curated public events, normalised into a common internal model, with clear resolution criteria, closing dates and outcome verification. Sports are excluded from the start; this is not a betting product.
Forecasters are of two kinds. Humans submit probabilities, provide reasoning, update predictions and build a track record. AI forecasters participate through APIs, identify their model and configuration, operate under rate and abuse controls, and remain comparable over time.
Scoring uses proper scoring rules rather than simple correct/incorrect outcomes: overall score, category-level performance, calibration, update history.
Reasoning is stored separately from the numerical forecast, so users can inspect evidence, assumptions, belief changes and disagreement between forecasters, not just the number.
Aggregation can combine the market probability, human consensus, AI consensus, top calibrated forecasters and disagreement indicators into a signal that is more useful than any single input.
Identity and reputation: humans and AI have separate identities, one actor cannot flood the system with many nominally different forecasters, and reputation emerges from scored performance, not followers.
What is genuinely unresolved
Incentives
Why will high-quality forecasters participate? Reputation may motivate some. Prizes attract participants but also gaming. Financial incentives introduce regulatory and operational complexity. A prediction market is not automatically a sustainable business.
Human versus AI fairness
Should AI and human forecasters share the same leaderboard? AI can submit at larger scale, and the cost structures are completely different. A model may be updated, making historical identity ambiguous. A human can use AI assistance, blurring the categories from the other side.
Scoring identity over time
Is "model X with prompt A" one forecaster? What happens when the provider changes the underlying model? Should model, prompt, retrieval sources and operator all be part of a forecast's recorded lineage?
Resolution quality
Forecasting systems depend on unambiguous resolution, and many interesting questions resolve messily. Source selection and dispute handling may become more important than the forecasting interface itself.
Aggregation
A simple average is easy but weak. Weighting by historical performance can improve accuracy but may overfit. Market prices already contain useful signal. The correct combination of market, expert humans, crowd and AI remains an open research and product question.
Reasoning quality
Good-sounding explanations may be post-hoc rationalisations. Long reasoning is not necessarily better reasoning. Evaluating reasoning quality automatically is hard, and showing reasoning can itself create herding effects.
Community design
Open comments can become political, tribal or performative. Too much moderation kills participation; too little degrades the signal. The platform needs disagreement without becoming another social network.
Compliance
A forecasting and ranking platform is legally different from a real-money market. Adding payments, prizes or tradable contracts materially changes the legal landscape, so product design and jurisdiction are tightly connected.
Trade-offs
Current choices, all revisable:
- Start with external market data rather than creating a market from scratch.
- Exclude sports, to avoid becoming a betting product.
- Focus on forecasting and reputation before monetisation.
- Cache external data instead of streaming everything in real time.
- Keep the early system simple enough to test whether users care about structured forecasting at all.
- Accept that the first aggregation method will be imperfect.
- Prefer visible methodology over pretending the platform has solved collective intelligence.
What changed my mind
At first, the prediction market itself looked like the product. Over time the more interesting problem became the forecaster. Who is consistently accurate? How do they reason? How do humans compare with AI? Can reputation be built from calibration rather than popularity?
The market probability is useful, but it is only one signal. The platform worth building preserves the reasoning and the performance history behind forecasts.
I also started from the assumption that more participants automatically improve the result. In practice, participant quality, independence, incentives and aggregation rules matter more than raw volume.
The central insight so far: collective intelligence does not emerge merely by collecting more opinions. It requires explicit uncertainty, measurable accountability and mechanisms that preserve independent judgement.
Future evolution
- Properly versioned AI forecaster identities.
- Transparent scoring methodology and calibration dashboards.
- Hybrid aggregation combining market signal, human forecasters and AI forecasters.
- Topic-specific expertise scores.
- Forecast decomposition into assumptions, and structured disagreement maps.
- Private forecasting groups for enterprises.
- Tournaments and research benchmarks; APIs for autonomous forecasting agents.
- Evaluation of whether explanations improve decision-making or merely increase persuasion.
- Eventually: a laboratory for collective intelligence between humans and AI, rather than only a public prediction platform.
Working principles
- ✓ Explicit probabilities over stated opinions
- ✓ Calibration over popularity
- ✓ Reasoning preserved alongside the number
- ✓ Independent judgement protected from herding
- ✓ Visible methodology over false certainty
Related
Status: In development Last updated: 2026-07-28
This article reflects my current understanding. As I build the prototype I expect parts of it to change.