Calibration › Summary

Calibration measures the alignment between a model's predicted confidence and its actual accuracy. A perfectly calibrated model satisfies the condition that among all predictions where the model claims confidence $p$, exactly fraction $p$ are correct. This is a statistical property that applies to the model's behavior over many predictions, not to individual outputs.

The Expected Calibration Error quantifies miscalibration by binning predictions and measuring the weighted average gap between confidence and accuracy across bins. The weights reflect the fraction of predictions in each bin, so that densely populated confidence regions contribute more to the overall metric. MCE captures worst-case miscalibration in any single bin, which matters more in safety-critical applications. Both metrics depend on the binning strategy, and comparison across implementations requires care.

Neural networks systematically tend toward overconfidence due to the asymmetric gradient of cross-entropy loss, increased model capacity, and interactions with architectural components like batch normalization. Label smoothing during training can partially mitigate this tendency by preventing logits from growing without bound toward infinite confidence.

Temperature scaling provides a simple, effective post-hoc calibration method that divides logits by a learned scalar $T > 1$ to soften overconfident distributions. It preserves prediction rankings and achieves surprisingly competitive calibration performance given its single parameter. Isotonic regression offers a non-parametric alternative that can capture complex calibration curves but requires more data. Platt scaling and Beta calibration provide intermediate options with two to four parameters. It offers more flexibility than temperature scaling while being less prone to overfitting than isotonic regression.

Reliability diagrams visualize calibration by plotting accuracy against confidence at each binned confidence level, revealing overconfidence (points below the diagonal) or underconfidence (points above). Modern large language models achieve high accuracy but often suffer from overconfidence, making calibration an essential consideration for applications that rely on uncertainty estimates for decision-making, selective prediction, or human-AI collaboration.