probability-calibration · shipping · also probabilistic calibration, reliability of probability forecasts

Probability calibration

Probability calibration is agreement between forecast probabilities and conditional event frequencies among observations receiving those probabilities.

A reliability diagram groups forecast scores and compares mean probability with the observed positive fraction; report the group boundaries and counts. This estimates calibration on an evaluation sample, not a guarantee for every individual drive. [1]

A fitted calibrator must be evaluated without reusing its fitting data as the final test set. Guo et al. discuss held-out calibration and temperature scaling in image and document classification, not storage failure prediction. [2]

Relations

related

brier-score, average-precision, matthews-correlation-coefficient, prediction-horizon, temporal-validation, class-prevalence, expected-calibration-error

Limitations

• Calibration alone does not establish useful ranking: a constant forecast can match a pooled event rate while distinguishing no devices; the linked example uses invented counts only. [3]

Citations

1. scikit-learn: probability calibration, reliability diagrams and limits of Brier loss. (accessed 2026-10-17)
2. Guo, Pleiss, Sun and Weinberger. On Calibration of Modern Neural Networks. ICML 2017, sections 4-4.2. (accessed 2026-10-17)
3. Hesela: synthetic ranking, decision and calibration example; not measured drive results. (accessed 2026-10-17)

JSON: /data/terms/probability-calibration.json