The Economist Benchmark Simulator

Human vs Machine Forecast Ensemble Lab

AI models now rival top superforecasters on geopolitics and macroeconomic predictions. Test calibrated ensembling, quantify Brier skill scores, and simulate the tipping point where synthetic intelligence overtakes human wisdom.

Ensemble Controls

12 Resolved Items

Benchmark Scenario Curated sets derived from Metaculus, Good Judgment Open, and LLM forecast benchmarks.

Ensemble Aggregation Model

Tetlock & Baron log-odds extremization corrects for crowd underconfidence.

Human vs Machine Weight 50% H / 50% AI

0% = Pure AI baseline • 100% = Pure Human Superforecaster crowd.

Extremization Multiplier (d) 1.40×

Pushes consensus towards 0 or 1 to reward independent signal overlap.

Simulate AI Capability Leap +0% (Baseline)

Sharpen AI probability resolution & reduce hallucinated uncertainty.

Human Forecasters 0.142 Brier Score (Lower is better)

Machine (LLM Agent) 0.138 Brier Score

Hybrid Ensemble 0.112 -21.1% error

Ensemble Skill Score +0.552 vs Climatology baseline

Reliability Diagram (Calibration Curve)

Perfect forecasting lies directly on the 45° dashed diagonal. Deviations reveal systematic over- or under-confidence.

Human

Machine

Hybrid Ensemble

Resolved Questions in Benchmark (12 items)

Observed Outcome (0 = Did not occur, 1 = Occurred)

Question	Outcome	Human Crowd	Machine LLM	Ensemble

Calibration updated. Hybrid model outperforms both standalone agents.

The Brier Score & Calibration

Developed by Glenn W. Brier in 1950, the Brier score measures the accuracy of probabilistic forecasts:

Brier Score = (1 / N) ∑ (f_i - o_i)²

Where f_i is the predicted probability [0, 1] and o_i is the binary outcome ∈ {0, 1}. Lower scores indicate superior skill, with 0.0 being clairvoyant perfection and 0.25 representing pure random guessing on balanced events.

The Brier score decomposes neatly into Reliability (calibration alignment), Resolution (ability to sort outcomes), and Uncertainty (intrinsic variance of reality).

Why Hybrid Ensembles Win

Human superforecasters excel at narrative nuance, insider intuition, and spotting structural breaks. LLM multi-agent pipelines excel at instantaneous historical base-rate retrieval, tireless news synthesis, and non-emotional Bayesian updating.

Log-Odds: φ(p) = ln(p / (1 - p))
Combined: p* = σ( d × [w × φ(p_H) + (1-w) × φ(p_M)] )

Because human and machine errors are partially uncorrelated, blending them in log-odds space with an extremization factor d > 1 eliminates mutual underconfidence and strips out single-agent bias.

Frequently Asked Questions

What was The Economist's finding regarding AI vs Human forecasters?

Recent empirical trials (including studies by Berkeley, Metaculus, and forecasting researchers) showed that state-of-the-art LLM ensembles (incorporating web retrieval, chain-of-thought debates, and median pooling) achieved Brier scores comparable to or exceeding the average crowd of human forecasters, and were on par with top superforecaster cohorts on multiple geopolitical horizons.

How does the "Log-Odds Extremized" algorithm work?

Standard linear probability averaging (e.g., (p1 + p2)/2) suffers from systematic conservatism because independent forecasters often hold complementary evidence that points in the same direction. Converting probabilities into log-odds space, taking a weighted combination, and then multiplying by an extremizing coefficient d (typically between 1.2 and 1.6) pulls consensus toward definitive confidence when both agents lean in agreement.

Are these calculations performed locally on my device?

Yes. All probability transforms, Murphy calibration binning, Brier score sums, and interactive sensitivity tests run locally in real-time within your browser engine. No data is sent to an external server.