{
  "status": "shipping",
  "advantages": [],
  "supersedes": [],
  "supersededBy": [],
  "firstSeen": null,
  "lastSeen": null,
  "updated": "2026-10-17",
  "slug": "expected-calibration-error",
  "term": "Expected calibration error",
  "aliases": [
    "ECE",
    "binned calibration error"
  ],
  "definition": "Expected calibration error is a summary of disagreement between predicted probabilities and conditional outcome frequencies, commonly estimated by bin-weighted gaps on an evaluation sample.",
  "note": "Specify the probability and outcome being scored. Predicted-class confidence versus classification correctness is not the same calculation as failure probability versus a binary failure label. Guo et al. use the former convention. [guo-icml17]\n\nFor binary-event L1 ECE, sum each bin's share of observations times the absolute difference between its mean probability and observed event fraction; publish edges and counts, not just the scalar. A fixed partition can conceal opposite errors within a bin. [vaicenavicius-aistats19]\n\nFinite-sample estimates can overestimate or underestimate population error; more bins are not an automatic remedy. The cited experiments concern classification benchmarks, not storage fleets. [roelofs-aistats22]",
  "limitations": [
    {
      "text": "Zero empirical binned error is not proof of calibrated failure probabilities, useful ranking, or transfer to future drives; the linked example uses stipulated outcomes.",
      "citationIds": [
        "hesela-binning"
      ]
    }
  ],
  "related": [
    "probability-calibration",
    "brier-score",
    "class-prevalence",
    "prediction-horizon",
    "grouped-cross-validation"
  ],
  "citations": [
    {
      "id": "guo-icml17",
      "title": "Guo et al. On Calibration of Modern Neural Networks. ICML 2017, section 2 and equation 3.",
      "url": "https://proceedings.mlr.press/v70/guo17a/guo17a.pdf",
      "accessed": "2026-10-17"
    },
    {
      "id": "vaicenavicius-aistats19",
      "title": "Vaicenavicius et al. Evaluating model calibration in classification. AISTATS 2019, sections 4.1-4.4 (fixed-bin asymptotics and estimator variability).",
      "url": "https://proceedings.mlr.press/v89/vaicenavicius19a/vaicenavicius19a.pdf",
      "accessed": "2026-10-17"
    },
    {
      "id": "roelofs-aistats22",
      "title": "Roelofs et al. Mitigating Bias in Calibration Error Estimation. AISTATS 2022, section 4 and Appendix B (finite-sample bias).",
      "url": "https://proceedings.mlr.press/v151/roelofs22a/roelofs22a.pdf",
      "accessed": "2026-10-17"
    },
    {
      "id": "hesela-binning",
      "title": "Hesela: synthetic binary-event binning counterexample and reproducible arithmetic.",
      "url": "https://hesela.com/analyses/ml-disk-failure-prediction/#calibration-binning",
      "accessed": "2026-10-17"
    }
  ]
}