Probability is a promise about frequency

You can be calibrated at 50% and still be less accurate.

A constant 50% forecast is calibrated when half the outcomes happen. It cannot distinguish a likely case from an unlikely one, which is where better forecasts gain accuracy.

Run a calibration check

Use one probability bucket and its observed outcomes. The calculation runs entirely in this browser.

Sample loaded: a perfectly calibrated 50% bucket.

Measured result

Calibration is closeness of the forecast to the observed frequency. Accuracy here is the share correct when each forecast is turned into its most likely binary answer.

Observed rate50.0%
Calibration gap0.0 pp
Binary accuracy50.0%
Brier score0.250
At 50%, 25 successes in 50 trials is calibrated. A rival can be more accurate by assigning higher probabilities to cases that happen and lower probabilities to cases that do not.

95% observed-rate interval: 36.6% to 63.4%.

Calibration is one axis.
Discrimination is another.

When every case gets 50%, the forecast makes the same decision every time. It can be calibrated across many cases, yet it leaves no room to rank cases. A more accurate forecaster can remain calibrated while using 70% for cases that happen often and 30% for cases that do not.

Calibration, accuracy and squared error answer different questions

Read the explanation

The saved example contains fifty forecasts at fifty percent and twenty five observed successes. Observed frequency is twenty five divided by fifty, or fifty percent. Bars share seven pixels per percentage point, so forecast and observed rate have equal lengths and zero calibration gap. With forty successes instead, the observed rate is eighty percent and the gap thirty points. These are entered counts, not a measured forecasting dataset; one bucket being calibrated does not establish discrimination across cases. The source turns probabilities at least fifty percent into a yes answer. For thirty five successes in fifty forecasts, the observed rate is seventy percent. A forecast of sixty percent predicts yes and has seventy percent binary accuracy; forty percent predicts no and has thirty percent accuracy. Bars share six pixels per percentage point. The threshold changes the answer without changing outcomes; this is not a claim that higher confidence guarantees a correct forecast. For a bucket with half successes, the source Brier score averages squared probability errors. At fifty percent the mean score is zero point two five. At ninety percent on the same half-success bucket it is zero point four one. Bars share one thousand pixels per score unit. Lower is better for this entered calculation: overconfidence in this hypothetical bucket increases squared error. The source also exports a Wilson observed-rate interval, not a guarantee about future forecasts. Trial and success counts are rounded and invalid inputs block calculation and export.

Super generates helpful tools and automates fact-checking across the internet proactively. If you enjoyed this tool, build your own with Super and share it with a friend.