Probability Calibration › Example

A small numerical example makes the decomposition concrete and shows why calibration can improve proper scores without changing the ranking. Suppose a model produces ten predictions and we group them into two reliability bins. Every case in the high-score bin receives probability 0.9; four of its five cases are positive. Every case in the low-score bin receives probability 0.6; one of its five cases is positive. Using only bin means would not determine the original unbinned scores. We fit and score the illustrative mapping on these same ten cases to show the arithmetic; the later implementation uses separate calibration and test partitions.

The raw Brier score is the average squared difference between each predicted probability and its outcome.

$$\begin{aligned} \mathrm{BS}_{\text{high}} &= 4(0.9-1)^2+1(0.9-0)^2 \\ &= 4(0.01)+0.81 \\ &= 0.85. \end{aligned}$$

In the low bin, one case has outcome 1 and four have outcome 0:

$$\begin{aligned} \mathrm{BS}_{\text{low}} &= 1(0.6-1)^2+4(0.6-0)^2 \\ &= 0.16+4(0.36) \\ &= 1.60. \end{aligned}$$

The overall Brier score is

$$\begin{aligned} \mathrm{BS} &= \frac{0.85+1.60}{10} \\ &= 0.245. \end{aligned}$$

Now we decompose it. The reliability term is

$$\begin{aligned} \mathrm{REL} &= \frac{5}{10}(0.9-0.8)^2 + \frac{5}{10}(0.6-0.2)^2 \\ &= 0.5(0.01)+0.5(0.16) \\ &= 0.085. \end{aligned}$$

The resolution term is

$$\begin{aligned} \mathrm{RES} &= \frac{5}{10}(0.8-0.5)^2 + \frac{5}{10}(0.2-0.5)^2 \\ &= 0.5(0.09)+0.5(0.09) \\ &= 0.09. \end{aligned}$$

The uncertainty term is

$$\begin{aligned} \mathrm{UNC} &= \bar y(1-\bar y) \\ &= 0.5(0.5) \\ &= 0.25, \end{aligned}$$

where $\bar y=0.5$ is the base rate, the fraction of positive cases in the full sample. So

$$\begin{aligned} \mathrm{REL}-\mathrm{RES}+\mathrm{UNC} &= 0.085-0.09+0.25 \\ &= 0.245, \end{aligned}$$

which matches the direct Brier score. The expected calibration error is

$$\begin{aligned} \mathrm{ECE} &= 0.5|0.8-0.9|+0.5|0.2-0.6| \\ &= 0.5(0.1)+0.5(0.4) \\ &= 0.25. \end{aligned}$$

The raw log loss is

$$\begin{aligned} \mathrm{LL} &= -\frac{1}{10} \left[ 4\log(0.9)+\log(0.1) + \log(0.6)+4\log(0.4) \right] \\ &\approx -\frac{1}{10}\left[4(-0.10536)+(-2.30259)+(-0.51083)+4(-0.91629)\right] \\ &\approx 0.690. \end{aligned}$$

Now apply an isotonic recalibration. The high bin maps to 0.8 and the low bin maps to 0.2, because those are the observed rates in the two monotone blocks. The calibrated Brier score becomes

$$\begin{aligned} \mathrm{BS}_{\text{cal}} &= \frac{ 4(0.8-1)^2+(0.8-0)^2 + (0.2-1)^2+4(0.2-0)^2 }{10} \\ &= 0.160. \end{aligned}$$

The calibrated log loss becomes

$$\begin{aligned} \mathrm{LL}_{\text{cal}} &= -\frac{1}{10} \left[ 4\log(0.8)+\log(0.2) + \log(0.2)+4\log(0.8) \right] \\ &\approx 0.500. \end{aligned}$$

Both proper scores improve substantially. The ranking, however, does not improve. There are four positive cases in the high bin and one positive case in the low bin, with one negative case in the high bin and four negative cases in the low bin. Pairing each positive with each negative gives 25 pairs. Predictions are tied within each bin both before and after this mapping. There are sixteen cross-bin pairs with the positive in the high bin and the negative in the low bin, four within-high-bin tied pairs, four within-low-bin tied pairs, and one cross-bin pair with the positive in the low bin and the negative in the high bin. Treating ties as one-half gives

$$\begin{aligned} \mathrm{AUC} &= \frac{16+0.5(4)+0.5(4)}{25} \\ &= \frac{20}{25} \\ &= 0.8. \end{aligned}$$

The calibrated scores preserve this ranking, so AUC remains 0.8. This example shows the central distinction: calibration repairs the probability scale, while discrimination is about the ordering. That is why we inspect both calibration and ranking.

The additive decomposition can be read as a signed account: reliability adds penalty, resolution subtracts a reward, and uncertainty is the empirical constant base-rate term. The next figure shows those terms for the two-bin example and checks that they add to the raw Brier score.