Calibration › Worked Example

We will walk through a concrete example to see calibration in action. Imagine a sentiment classifier that outputs probabilities for Positive and Negative classes. We evaluate it on 10 samples:

Examining this table, we see the model makes predictions across the confidence spectrum from 0.50 to 0.95. Out of 10 predictions, 7 are correct, giving an overall accuracy of 70%. However, the average confidence is higher than 70%, suggesting overconfidence. To quantify this miscalibration, we need to group these predictions into bins and compare the accuracy and confidence within each bin.

Using 5 equal-width bins: [0.5-0.6), [0.6-0.7), [0.7-0.8), [0.8-0.9), [0.9-1.0]

• Bin [0.5-0.6): 2 samples (0.55, 0.50), confidence = 0.525, accuracy = 0.5 (1 correct out of 2)
• Bin [0.6-0.7): 2 samples (0.65, 0.60), confidence = 0.625, accuracy = 1.0 (2 correct out of 2)
• Bin [0.7-0.8): 2 samples (0.75, 0.70), confidence = 0.725, accuracy = 0.5 (1 correct out of 2)
• Bin [0.8-0.9): 2 samples (0.85, 0.80), confidence = 0.825, accuracy = 0.5 (1 correct out of 2)
• Bin [0.9-1.0]: 2 samples (0.95, 0.90), confidence = 0.925, accuracy = 1.0 (2 correct out of 2)

Notice how the accuracy fluctuates relative to confidence. In the [0.6-0.7) bin, the model is underconfident, achieving perfect accuracy while expressing only 62.5% confidence. However, in the [0.7-0.8) and [0.8-0.9) bins, the model is overconfident, claiming 72.5% and 82.5% confidence respectively while only achieving 50% accuracy.

The ECE calculation proceeds by summing the weighted absolute gaps across all bins:

• Bin 1: $|0.5 - 0.525| \times 0.2 = 0.005$
• Bin 2: $|1.0 - 0.625| \times 0.2 = 0.075$
• Bin 3: $|0.5 - 0.725| \times 0.2 = 0.045$
• Bin 4: $|0.5 - 0.825| \times 0.2 = 0.065$
• Bin 5: $|1.0 - 0.925| \times 0.2 = 0.015$

Total ECE = 0.205 or 20.5%

This indicates significant miscalibration, particularly in the middle confidence ranges where the model is overconfident. The ECE of 20.5% suggests that, on average, the model's confidence deviates from its accuracy by over 20 percentage points across the confidence spectrum. An ECE of 20.5% is quite high; in practice, well-calibrated models achieve ECE values below 5%, and many production systems aim for ECE below 2%.

Notice also that the MCE in this example is the maximum bin gap, which is bin 2: $|1.0 - 0.625| = 0.375$. This means that in the confidence range [0.6-0.7), the model's accuracy is 37.5 percentage points higher than its confidence, indicating substantial underconfidence in that range. If this were a real model, you would want to understand why the model is so certain about its errors in the middle confidence range while being correct when expressing slightly lower confidence.

With only 10 samples, these estimates are highly noisy. Each bin contains only 2 predictions, making the accuracy estimate either 0.0, 0.5, or 1.0 with no intermediate values. This example illustrates why ECE estimation requires large datasets: with a typical real evaluation containing thousands of predictions, the accuracy estimates within each bin become much more reliable and the resulting ECE value more meaningful.