Calibration › Measuring Calibration

Since we cannot verify the exact equality $P(\hat{Y} = Y \mid \hat{P} = p) = p$ for continuous $p$, we measure calibration error by grouping predictions into bins based on their confidence scores. This discretization allows us to empirically estimate the relationship between predicted confidence and observed accuracy. Without binning, we would rarely see multiple predictions with exactly the same confidence value. This makes statistical estimation impossible. The binning strategy turns the continuous calibration problem into a discrete estimation task where we can compare average confidence against empirical accuracy within each bin.

The fundamental trade-off in binning is resolution versus reliability. With many fine-grained bins, you can detect localized calibration errors, but each bin contains few samples, making the accuracy estimate noisy. With few coarse bins, each estimate is more stable but may average over meaningful within-bin variation. There is no universally optimal number of bins; the choice depends on the total number of predictions available and how finely you need to characterize the calibration curve.

The Expected Calibration Error (ECE) is the standard metric for quantifying miscalibration. It partitions predictions into $M$ equally-spaced bins and computes the weighted average of the absolute difference between accuracy and confidence in each bin. This metric provides a single scalar value representing the overall calibration quality of the model, where lower values indicate better alignment between confidence and accuracy.

For bin $B_m$ containing $n_m$ predictions, we compute the empirical accuracy:

$$\text{acc}(B_m) = \frac{1}{n_m} \sum_{i \in B_m} \mathbb{1}(\hat{y}_i = y_i)$$

where:

• $B_m$: the set of predictions falling into bin $m$
• $n_m$: the number of predictions in bin $m$
• $\hat{y}_i$: the predicted label for the $i$-th sample
• $y_i$: the true label for the $i$-th sample
• $\mathbb{1}(\cdot)$: the indicator function, equal to 1 when the condition is true and 0 otherwise

We also compute the average confidence within the bin:

$$\text{conf}(B_m) = \frac{1}{n_m} \sum_{i \in B_m} \hat{p}_i$$

where:

• $\hat{p}_i$: the predicted confidence (probability) for the $i$-th sample

The ECE is then the weighted sum of absolute differences across all bins:

$$\text{ECE} = \sum_{m=1}^{M} \frac{n_m}{n} \left| \text{acc}(B_m) - \text{conf}(B_m) \right|$$

where:

• $M$: the total number of bins
• $n$: the total number of samples across all bins
• $\frac{n_m}{n}$: the proportion of samples in bin $m$, serving as the weight for that bin's contribution

The weighting by $\frac{n_m}{n}$ is critical and reflects a sensible design choice: calibration errors in densely populated bins matter more than errors in sparse ones. If a model is slightly off in a bin containing 2% of predictions, that matters far less than being off in a bin containing 40% of predictions. An ECE of 0 indicates perfect calibration; higher values indicate worse calibration. The absolute value ensures that overconfidence and underconfidence both count as errors rather than canceling out.

The choice of $M$ matters. Too few bins lose resolution and may mask local calibration errors by averaging over wide confidence ranges. Too many bins create noisy estimates with few samples per bin, leading to high variance in the accuracy estimates. Common choices are 10 or 15 bins, balancing granularity with statistical reliability. Research comparing different binning choices finds that the optimal number depends on dataset size: for datasets with thousands of predictions, 15 bins works well; for smaller datasets, 5 to 10 bins provide more stable estimates.

One subtle issue with ECE is that it measures the expected calibration error for the top-1 confidence. In multiclass settings with many classes, the model could be perfectly calibrated at the top-1 level while being systematically wrong about the probability mass assigned to non-top classes. If you care about the full distribution, such as when using the model's outputs as inputs to another system, you would need to measure classwise calibration separately for each class.

While ECE captures average calibration across all confidence levels, the Maximum Calibration Error (MCE) identifies the worst-case deviation:

$$\text{MCE} = \max_{m \in \{1, \ldots, M\}} \left| \text{acc}(B_m) - \text{conf}(B_m) \right|$$

where:

• $\max$: the maximum operator, selecting the largest absolute deviation between accuracy and confidence across all bins
• $m$: the bin index, ranging from $1$ to $M$
• $M$: the total number of bins (as defined previously for ECE)
• $\text{acc}(B_m)$: the accuracy of predictions in bin $m$ (fraction correct)
• $\text{conf}(B_m)$: the average confidence of predictions in bin $m$

MCE is particularly important in safety-critical applications where even occasional overconfidence can be dangerous. While a model might exhibit low average calibration error, a specific confidence range where the model is severely overconfident could lead to catastrophic failures. For instance, if a medical diagnosis model is perfectly calibrated at low and medium confidence but severely overconfident when predicting above 95% probability, clinicians might be misled into trusting incorrect high-confidence diagnoses. In such a system, having a small average ECE provides false reassurance: what matters is that the high-stakes high-confidence region is calibrated, even if it contains relatively few predictions overall.

The trade-off between ECE and MCE reflects a choice between optimizing for average behavior versus worst-case behavior. In most evaluation contexts, ECE is reported because it summarizes overall calibration quality. When deploying models in high-stakes settings, MCE deserves more attention because it reveals the most dangerous miscalibration that could occur.

Equal-width binning can leave some bins empty or sparsely populated if the model's confidence distribution is skewed. Many neural networks tend toward extreme confidence values, clustering predictions near 0 and 1, leaving middle ranges empty. Adaptive binning, also called equal-mass binning, ensures each bin contains approximately the same number of samples by adjusting bin boundaries based on the empirical distribution of confidence scores. This approach prevents empty bins and ensures that each accuracy estimate is computed over a sufficient sample size.

The procedure for adaptive binning is straightforward: sort all predictions by their confidence scores, then divide them into $M$ groups of equal size. The boundaries between bins are the confidence values at the $k/M$ quantiles for $k = 1, \ldots, M-1$. Every bin thus contains exactly $\lfloor n/M \rfloor$ or $\lceil n/M \rceil$ predictions. This ensures that the accuracy estimate in each bin is equally statistically reliable, a property that equal-width binning lacks when the confidence distribution is non-uniform.

The trade-off is interpretability. With equal-width binning, the bin for "predictions between 0.7 and 0.8" has an obvious semantic meaning. With adaptive binning, the boundaries are data-dependent and harder to communicate. For model comparison, adaptive binning is often more reliable; for explaining calibration behavior to non-technical stakeholders, equal-width binning is easier to explain. This often provides more reliable ECE estimates, especially for models that are rarely uncertain or rarely highly confident.

Beyond computing a single ECE number, it is informative to examine the full calibration curve through a reliability diagram. The ECE aggregates calibration errors into a single number, necessarily losing information about where in the confidence spectrum the errors occur. Two models with identical ECE values can have very different calibration profiles: one might be overconfident at medium confidence levels but perfectly calibrated at high confidence, while another shows the opposite pattern. Understanding the shape of the miscalibration helps choose an appropriate correction method.

When examining reliability diagrams, look for these patterns:

• Uniform overconfidence: The curve sits below the diagonal at all confidence levels. This is the most common pattern in neural networks and responds well to temperature scaling.
• Non-uniform overconfidence: The curve crosses the diagonal, being overconfident in some ranges and underconfident in others. This pattern requires a non-parametric correction like isotonic regression.
• Confidence collapse: Most predictions cluster near 0.5, indicating the model is uncertain about nearly everything. This can happen with under-trained models or models facing extreme out-of-distribution inputs.
• Extreme overconfidence: Predictions cluster near 1.0 regardless of true accuracy. This is common in models trained with very large datasets where the model has memorized training examples and generalizes that certainty to test data inappropriately.