expected-calibration-error · shipping · also ECE, binned calibration error

Expected calibration error

Expected calibration error is a summary of disagreement between predicted probabilities and conditional outcome frequencies, commonly estimated by bin-weighted gaps on an evaluation sample.

Specify the probability and outcome being scored. Predicted-class confidence versus classification correctness is not the same calculation as failure probability versus a binary failure label. Guo et al. use the former convention. [1]

For binary-event L1 ECE, sum each bin's share of observations times the absolute difference between its mean probability and observed event fraction; publish edges and counts, not just the scalar. A fixed partition can conceal opposite errors within a bin. [2]

Finite-sample estimates can overestimate or underestimate population error; more bins are not an automatic remedy. The cited experiments concern classification benchmarks, not storage fleets. [3]

Relations

related

probability-calibration, brier-score, class-prevalence, prediction-horizon, grouped-cross-validation

Limitations

• Zero empirical binned error is not proof of calibrated failure probabilities, useful ranking, or transfer to future drives; the linked example uses stipulated outcomes. [4]

Citations

1. Guo et al. On Calibration of Modern Neural Networks. ICML 2017, section 2 and equation 3. (accessed 2026-10-17)
2. Vaicenavicius et al. Evaluating model calibration in classification. AISTATS 2019, sections 4.1-4.4 (fixed-bin asymptotics and estimator variability). (accessed 2026-10-17)
3. Roelofs et al. Mitigating Bias in Calibration Error Estimation. AISTATS 2022, section 4 and Appendix B (finite-sample bias). (accessed 2026-10-17)
4. Hesela: synthetic binary-event binning counterexample and reproducible arithmetic. (accessed 2026-10-17)

JSON: /data/terms/expected-calibration-error.json