Probability Calibration › Formula

Calibration is easiest to understand as a promise about conditional frequency. Imagine collecting every case for which the model says 0.30 and asking what fraction of those cases turned out positive. A calibrated model keeps that promise: the fraction should be about 0.30. To state the promise formally, let the binary outcome be $Y \in \{0,1\}$, and let a model produce a score $p_i$ for observation $i$. We treat $p_i$ as a claimed probability that $Y_i=1$. Population calibration means that the conditional positive probability equals the forecast, almost surely under the forecast distribution. Among independently sampled cases from the target population, a finite group's observed event fraction fluctuates around that probability:

$$\Pr(Y=1 \mid p_{\text{pred}}=p)=p \quad \text{for almost every } p \text{ under the forecast distribution.}$$

where:

• $Y$: the binary outcome, with $Y=1$ for a positive case and $Y=0$ for a negative case
• $p_{\text{pred}}$: the model's predicted probability for a case
• $p$: a specific predicted-probability value at which we inspect the conditional positive rate

This condition applies across the probability scale. Equivalently, for a calibrated forecast $P$,

$$\mathbb{E}[Y \mid P]=P \quad \text{almost surely}.$$

where:

• $Y$: the binary outcome
• $P$: the calibrated forecast probability, treated as a random variable
• $\mathbb{E}[Y \mid P]$: the conditional positive rate as a function of the forecast $P$

Taking expectations over the forecast distribution gives a useful consequence:

$$\mathbb{E}[Y]=\mathbb{E}\big[\mathbb{E}[Y\mid P]\big]=\mathbb{E}[P].$$

where:

• $\mathbb{E}[Y]$: the overall positive rate (base rate) of the population
• $\mathbb{E}[Y\mid P]$: the conditional positive rate given forecast $P$
• $\mathbb{E}[P]$: the average predicted probability across the population

So a calibrated model must have average predicted probability equal to the base rate. This is necessary but not sufficient. A constant predictor that always outputs the base rate is calibrated, but it has no discrimination.

A model can be useful in two different ways, and calibration addresses only one of them. Calibration asks whether the probability scale matches event rates: among cases assigned 0.8, is the positive-event rate about 80 percent? Discrimination asks whether the model separates positives from negatives: do positives tend to receive higher scores than negatives? AUC measures ranking:

$$\mathrm{AUC}=\Pr\big(s(X^{+})>s(X^{-})\big) +\frac12\Pr\big(s(X^{+})=s(X^{-})\big),$$

Here $X^{+}$ and $X^{-}$ are independent draws from the positive and negative populations, both of which must occur, and $s$ is the score. AUC gives half credit to a tie. It depends only on score order and ties, so any strictly increasing transformation of the scores leaves AUC unchanged. A model can have high AUC and poor calibration. For example, it might assign scores near 0.99 to positives and near 0.98 to negatives. The ranking is nearly perfect, but the probabilities are not trustworthy. Conversely, the constant population base-rate forecast is calibrated and has AUC equal to 0.5 when both classes occur.

If we want a single number to reward honest probabilities, we need a loss whose best possible report is the truth. That is the idea of a proper scoring rule. For a binary outcome with true probability $p$, a scoring rule $S(q,y)$ is proper if the following inequality holds for every reported probability $q$:

$$\mathbb{E}_{Y\sim \mathrm{Bernoulli}(p)}[S(q,Y)] \ge \mathbb{E}_{Y\sim \mathrm{Bernoulli}(p)}[S(p,Y)]$$

where:

• $S(q,Y)$: the score assigned to reported probability $q$ under outcome $Y$
• $p$: the true probability that $Y=1$
• $q$: the reported probability we are evaluating
• $\mathbb{E}_{Y\sim \mathrm{Bernoulli}(p)}[\cdot]$: the expectation over outcomes drawn from a Bernoulli distribution with success probability $p$

For a strictly proper rule, equality holds only when $q=p$. The Brier score is the mean squared error of the probabilities:

$$\mathrm{BS} = \frac{1}{n}\sum_{i=1}^{n}(p_i-y_i)^2.$$

where:

• $n$: the number of cases
• $p_i$: the predicted probability for case $i$
• $y_i$: the observed outcome for case $i$, coded as 1 for positive and 0 for negative

To see why it is proper, fix a true probability $p$ and suppose we report $q$. The expected squared error is

$$\begin{aligned} \mathbb{E}[(q-Y)^2\mid p] &= p(1-q)^2+(1-p)q^2 \\ &= p(1-2q+q^2)+(1-p)q^2 \\ &= p - 2pq + pq^2 + q^2 - pq^2 \\ &= (q-p)^2+p(1-p). \end{aligned}$$

where:

• $q$: the reported probability
• $Y$: the outcome, which is 1 with probability $p$ and 0 with probability $1-p$
• $p$: the true probability that $Y=1$

The second term does not depend on $q$, and the first term is minimized at $q=p$. Log loss is the negative log likelihood:

$$\mathrm{LL} = -\frac{1}{n}\sum_{i=1}^{n} \left[ y_i\log p_i+(1-y_i)\log(1-p_i) \right].$$

For a true probability $p$ and reported $q$, the expected log loss is

$$-p\log q-(1-p)\log(1-q).$$

where:

• $p$: the true probability that $Y=1$
• $q$: the reported probability

For $0<p<1$ and $0<q<1$, differentiating with respect to $q$ gives

$$\begin{aligned} -\frac{p}{q}+\frac{1-p}{1-q}&=0 \\ \Longleftrightarrow\quad -p(1-q)+(1-p)q&=0 \\ \Longleftrightarrow\quad -p+pq+q-pq&=0 \\ \Longleftrightarrow\quad q&=p. \end{aligned}$$

The minimum is unique because the second derivative is $p/q^2+(1-p)/(1-q)^2>0$ in the interior. For $p=0$ or $p=1$, the minimum occurs at the matching endpoint, using the usual $0\log0=0$ limit. Log loss therefore has the same properness property. It penalizes confident mistakes much more strongly than the Brier score because the logarithm is unbounded near zero. Thresholded accuracy cannot identify the true probability: many reports on the same side of the threshold give the same label. At threshold one half, the corresponding zero-one loss is weakly proper under the definition above, but it is not strictly proper. AUC is a pairwise ranking measure rather than a per-observation Bernoulli loss $S(q,y)$, and it does not identify the numerical probability scale.

The properness condition has a visual signature: for a fixed true probability, the expected Brier score is smallest when the reported probability equals the truth. The next figure draws that expected loss for three true probabilities and marks each minimum.

The formulas above describe an ideal conditional frequency. In practice we estimate that frequency by grouping predictions into bins and comparing the average prediction in each bin with the observed positive rate. A reliability diagram is the graphical version of that comparison, and it is often the first diagnostic we inspect. Suppose bin $b$ contains $n_b$ cases, with mean predicted probability

$$\mathrm{conf}_b=\frac{1}{n_b}\sum_{i\in b}p_i$$

and observed positive frequency

$$\mathrm{acc}_b=\frac{1}{n_b}\sum_{i\in b}y_i.$$

where:

• $b$: the index of the reliability bin
• $n_b$: the number of cases in bin $b$
• $p_i$: the predicted probability for case $i$
• $y_i$: the observed outcome for case $i$
• $\mathrm{conf}_b$: the mean predicted positive-event probability in bin $b$
• $\mathrm{acc}_b$: the observed positive rate in bin $b$, rather than thresholded classification accuracy

We plot $\mathrm{conf}_b$ on the horizontal axis and $\mathrm{acc}_b$ on the vertical axis. The diagonal is perfect calibration. A point below the diagonal means the model overpredicts the positive-event rate; a point above means it underpredicts that rate. Calling either direction overconfidence across the whole binary probability scale would confuse positive-event risk with confidence in the more likely class. The number of bins and the binning strategy matter: equal-width bins can put very few observations in extreme bins, while equal-frequency bins can hide poor calibration near the extremes. A reliability diagram is a diagnostic, not a single-number score.

The expected calibration error summarizes the same binned comparison:

$$\mathrm{ECE} = \sum_{b=1}^{B} \frac{n_b}{n} \left| \mathrm{acc}_b-\mathrm{conf}_b \right|.$$

where:

• $B$: the number of bins
• $n_b$: the number of cases in bin $b$
• $n$: the total number of cases
• $\mathrm{acc}_b$: the observed positive rate in bin $b$
• $\mathrm{conf}_b$: the mean predicted probability in bin $b$

ECE depends on the binning scheme. Merging bins can cancel oppositely signed gaps and reduce ECE, even when the individual forecasts are unchanged. Tied scores can also make quantile-bin counts unequal or reduce the number of distinct bins. It should not be used as the sole calibration metric. For binned forecasts, where each case in bin $b$ is represented by the bin mean $\mathrm{conf}_b$, the Brier score decomposes as

$$\mathrm{BS}_{\text{bin}} = \mathrm{REL}-\mathrm{RES}+\mathrm{UNC},$$

where:

• $\mathrm{BS}_{\text{bin}}$: the Brier score of the binned forecasts
• $\mathrm{REL}$: the reliability term, the calibration penalty
• $\mathrm{RES}$: the resolution term, the discrimination reward
• $\mathrm{UNC}$: the score of the empirical constant base-rate forecast

and the reliability term is

$$\mathrm{REL} = \frac{1}{n}\sum_{b=1}^{B}n_b \left(\mathrm{conf}_b-\mathrm{acc}_b\right)^2,$$

the resolution term is

$$\mathrm{RES} = \frac{1}{n}\sum_{b=1}^{B}n_b \left(\mathrm{acc}_b-\bar y\right)^2,$$

where:

• $n_b$: the number of cases in bin $b$
• $n$: the total number of cases
• $\mathrm{acc}_b$: the observed positive rate in bin $b$
• $\bar y$: the overall base rate, the fraction of positive cases

and the uncertainty term is

$$\mathrm{UNC}=\bar y(1-\bar y).$$

This identity is exact for the forecasts after every score in a bin is replaced by that bin's mean. It need not equal the Brier score of the original varying scores. Reliability is a penalty: it is large when predicted and observed frequencies disagree. Resolution is large when bins have event rates far from the overall rate. It measures this bin-level refinement, rather than ROC-AUC. Uncertainty is the Brier score of the empirical constant base-rate forecast on these observations. A useful summary is the Brier skill score relative to the base-rate forecast:

$$\mathrm{BSS} = 1-\frac{\mathrm{BS}}{\mathrm{BS}_{\text{ref}}},$$

where:

• $\mathrm{BSS}$: the Brier skill score relative to the base-rate forecast
• $\mathrm{BS}$: the Brier score of the model being evaluated
• $\mathrm{BS}_{\text{ref}}$: the Brier score of the reference base-rate forecast, required to be positive

BSS is undefined when the reference score is zero. A reference using the evaluation sample's own base rate is a descriptive benchmark; a deployable constant forecast must be estimated before final evaluation.

Once the reliability diagram shows miscalibration, the next question is how to repair the probability scale without discarding the model. The repair is a mapping from raw score to calibrated probability, learned on held-out data. Sigmoid recalibration, often called Platt scaling, fits a two-parameter monotone map from the raw score to a calibrated probability. When the raw model outputs a probability $0<p_i<1$, we can work on its logit scale:

$$z_i=\operatorname{logit}(p_i)=\log\frac{p_i}{1-p_i}.$$

where:

• $p_i$: the raw probability output by the base model for case $i$
• $z_i$: the logit of that probability, which is the input scale for sigmoid recalibration
• $\operatorname{logit}(\cdot)$: the log-odds transformation

The calibrated probability is

$$q_i=\sigma(a z_i+b) = \frac{1}{1+e^{-(a z_i+b)}}.$$

where:

• $q_i$: the calibrated probability for case $i$
• $z_i$: the logit of the raw model probability
• $a$: the slope parameter on the logit scale
• $b$: the intercept parameter on the logit scale
• $\sigma(\cdot)$: the logistic sigmoid function

The parameters $a$ and $b$ are fitted on the calibration set by minimizing log loss:

$$\ell(a,b) = -\sum_{i=1}^{n} \left[ y_i\log q_i+(1-y_i)\log(1-q_i) \right].$$

where:

• $\ell(a,b)$: the negative log likelihood on the calibration set as a function of the slope and intercept
• $n$: the number of calibration cases
• $q_i$: the calibrated probability for case $i$, equal to $\sigma(a z_i+b)$
• $y_i$: the observed outcome for case $i$

The derivative of the logistic sigmoid satisfies $\sigma'(\eta)=\sigma(\eta)(1-\sigma(\eta))$, and the derivative of $\log\sigma(\eta)$ is $1-\sigma(\eta)$. Applying the chain rule gives

$$\frac{\partial \ell}{\partial a} = \sum_{i=1}^{n}(q_i-y_i)z_i, \qquad \frac{\partial \ell}{\partial b} = \sum_{i=1}^{n}(q_i-y_i).$$

where:

• $\ell$: the negative log likelihood from above
• $q_i$: the calibrated probability for case $i$
• $y_i$: the observed outcome for case $i$
• $z_i$: the logit of the raw model probability for case $i$

Setting these derivatives to zero gives the first-order conditions. The general first-order system has no direct closed-form solution, so we use a numerical optimizer such as BFGS. With separated calibration labels, an unregularized finite optimum need not exist. When $a=1$ and $b=0$, the map is the identity. With $b=0$ and $0<a<1$, the map moves probabilities toward one half; with $b=0$ and $a>1$, it moves them toward the endpoints. A nonzero intercept shifts the curve. Positive $a$ preserves order, negative $a$ reverses it, and $a=0$ gives a constant. Our unconstrained fit must therefore be checked before invoking an order-preservation claim. If the raw score is a margin rather than a probability, as with a support vector machine, the same idea applies directly as $q=\sigma(a s+b)$.

Isotonic recalibration is nonparametric. It seeks a nondecreasing function $g$ that minimizes

$$\sum_{i=1}^{n}\left(y_i-g(p_i)\right)^2 \quad\text{subject to}\quad p_i\le p_j \Longrightarrow g(p_i)\le g(p_j).$$

where:

• $g(\cdot)$: the fitted nondecreasing mapping from raw score to calibrated probability
• $g(p_i)$: the fitted calibrated probability for case $i$
• $p_i$: the raw score for case $i$
• $y_i$: the observed outcome for case $i$
• $n$: the number of calibration cases
• $p_i\le p_j \Longrightarrow g(p_i)\le g(p_j)$: the monotonicity constraint ensuring a higher raw score does not produce a lower calibrated probability

The monotonicity constraint encodes the reasonable requirement that a higher raw score should not produce a lower calibrated probability. The pool adjacent violators algorithm solves this problem. We first aggregate equal scores so they receive the same fitted value, sort the distinct scores, and merge adjacent weighted blocks whenever their means violate monotonicity. The fitted value for each block is its weighted mean outcome. PAVA gives block-constant fitted values at the observed scores. Our predictor linearly interpolates between those score thresholds; any monotone extension matching the fitted points has the same training objective. Isotonic regression is flexible and can correct complex miscalibration, but it requires more calibration data and can overfit. Outside the range of observed scores it must extrapolate, typically by holding the boundary value constant.

Both methods are post hoc. They use the calibration set to learn a mapping and then apply that mapping to new scores. They leave the base model fixed and use only its score. The mapping can alter ties or score orientation, but supplies no new feature information.

A calibrated model satisfies $\mathbb{E}[P]=\Pr(Y=1)$. This follows from the law of total expectation. Calibration is not preserved by arbitrary transformations: if $P$ is calibrated and we apply a nonlinear function $h$, the transformed value $h(P)$ need not be calibrated. Discrimination is different: a strictly increasing transformation preserves both ranking and existing ties, hence AUC. Finite-precision saturation or clipping can introduce extra ties.

Proper scoring rules assess the numerical probability forecast. A model can improve its Brier score by becoming better calibrated, by becoming better at separating classes, or both. For the constant population forecast $\pi=\Pr(Y=1)$, the expected Brier score is $\pi(1-\pi)$. A finite-sample score need not equal this expectation. A model with strong discrimination but poor calibration can have a worse Brier score than a calibrated model with weaker discrimination. This is why we report calibration diagnostics and proper scores together rather than relying on a single number.