Decision-making glossary
Brier score
The Brier score is a measure of how accurate probability forecasts are: the average of the squared differences between each forecast probability and what happened, counted as 1 if the event occurred and 0 if not. For yes-or-no forecasts it runs from 0, perfect, to 1. Glenn Brier introduced it in 1950 for weather forecasts.
For a single yes-or-no forecast the score is (p − o)², where p is the probability given and o is 1 or 0. Saying 90% for something that happens scores 0.01. Saying 90% for something that does not happen scores 0.81. Averaged over many forecasts, the score rewards being confident and right, and punishes confident misses heavily.
A score means little without a baseline. Always forecasting 50% scores 0.25 whatever happens. A better comparison is the Brier skill score, which measures improvement over a reference forecast, such as always predicting the base rate: positive is better than the reference, zero is no skill, negative is worse.
Small samples mislead. A handful of forecasts can produce a very good or very bad score by luck, and the score alone does not explain why. It can be split into reliability, resolution and uncertainty, which separates being well calibrated from being able to tell likely events from unlikely ones.
In Decize: Decize reports a Brier score once a workspace has five resolved forecasts, labels it an early sample below 20, and shows skill against the base rate beside it. See how →
Sources: Glenn W. Brier, “Verification of Forecasts Expressed in Terms of Probability”, Monthly Weather Review (1950) · WMO Joint Working Group on Forecast Verification Research, “Forecast verification: methods, issues and FAQ”