Probability Calibration

A classifier that returns predict_proba gives us estimated probabilities, but those estimates are not automatically calibrated. A model may say 0.90 for a group of cases in which only 70 percent are positive. The ranking may still be excellent, and the classifier may still be useful for sorting cases, but the number 0.90 cannot be read as a 90 percent chance. Probability calibration is the problem of making predicted probabilities match observed frequencies.

Many decisions use the probability itself. A fraud team may set a review threshold from expected costs, a hospital may use a risk score to allocate follow-up care, and a lending model may be monitored by comparing predicted default rates with realized default rates. In each case, a score of 0.2 should mean roughly a 20 percent event rate, not merely “lower than a score of 0.3.” Calibration is also distinct from discrimination. A model can rank positives above negatives very well while being badly overconfident, and a model can be perfectly calibrated while having no ability to separate the two classes.

Calibration is a property of the model and the population together, not a property of the model alone. A calibrator learned on one population can fail on another. We therefore treat calibration as a monitored, evidence-based step: fit it on representative held-out data, evaluate it on untouched data, and check whether the reliability holds across important subgroups and over time.

Calibrated probabilities can be used directly in expected-value calculations. If a predicted probability is 0.08 and the loss from a missed positive case is much larger than the cost of an unnecessary review, we can compare expected costs rather than guessing at an arbitrary threshold. This is the practical reason calibration matters in credit, insurance, medicine, fraud, churn, and maintenance. For a decision based on the score, calibration supports expected-cost calculations under the stated cost model. It does not establish each individual's risk given every feature, or justify combining probabilities without assumptions about their dependence.

Post-hoc calibration can be applied to many classifiers without retraining them. It learns a mapping from one raw score to an estimated probability using a separate calibration sample. This works for naive Bayes, support vector machines, random forests, boosted trees, and neural networks. It does not require retraining the original model, and it can be applied after model selection. Conditional-independence violations can make naive Bayes scores too extreme. A fitted recalibrator may improve their probability quality, but that must be checked on untouched data.

Calibration also supports communication and monitoring. A reliability diagram gives a direct visual answer to the question “when the model says 30 percent, how often is the event true?” Proper scores such as Brier score and log loss summarize probability quality in a single number. These diagnostics make it easier to compare models on the scale that decisions use, and they expose failures that accuracy and AUC can hide.

Calibration requires held-out data that are representative of the population where the probabilities will be used. If the calibration set is small, the fitted mapping is noisy, especially in the tails. Isotonic recalibration has flexible monotone fitted values at the observed scores; our predictor interpolates between those scores. With few observations, it can overfit the calibration sample and produce unstable probabilities on new cases. The two-parameter sigmoid limits flexibility, which can help with a smaller sample, but it can underfit complex patterns and still be poorly estimated.

A strictly increasing recalibration map preserves score order and ROC-AUC. Isotonic maps can create ties, which can change AUC substantially in either direction. An unconstrained sigmoid can also reverse order if its fitted slope is negative. These mappings use only the original score; they cannot recover feature information that score has discarded. That mapping leaves the base fit and its training labels unchanged. With clean, representative calibration labels, it can still correct some probability-scale distortions caused by label noise or a nonrepresentative training sample.

Calibration is also population-specific. A mapping that is valid on one hospital, region, customer segment, or time period may be invalid on another. Under distribution shift, the relationship between predicted probability and outcome can change, and a previously calibrated model can become miscalibrated. For these reasons, calibration should be checked on untouched data, across subgroups, and over time rather than assumed to hold permanently.

Calibration is easiest to understand as a promise about conditional frequency. Imagine collecting every case for which the model says 0.30 and asking what fraction of those cases turned out positive. A calibrated model keeps that promise: the fraction should be about 0.30. To state the promise formally, let the binary outcome be $Y \in \{0,1\}$, and let a model produce a score $p_i$ for observation $i$. We treat $p_i$ as a claimed probability that $Y_i=1$. Population calibration means that the conditional positive probability equals the forecast, almost surely under the forecast distribution. Among independently sampled cases from the target population, a finite group's observed event fraction fluctuates around that probability:

where:

• $Y$: the binary outcome, with $Y=1$ for a positive case and $Y=0$ for a negative case
• $p_{\text{pred}}$: the model's predicted probability for a case
• $p$: a specific predicted-probability value at which we inspect the conditional positive rate

This condition applies across the probability scale. Equivalently, for a calibrated forecast $P$,

where:

• $Y$: the binary outcome
• $P$: the calibrated forecast probability, treated as a random variable
• $\mathbb{E}[Y \mid P]$: the conditional positive rate as a function of the forecast $P$

Taking expectations over the forecast distribution gives a useful consequence:

where:

• $\mathbb{E}[Y]$: the overall positive rate (base rate) of the population
• $\mathbb{E}[Y\mid P]$: the conditional positive rate given forecast $P$
• $\mathbb{E}[P]$: the average predicted probability across the population

So a calibrated model must have average predicted probability equal to the base rate. This is necessary but not sufficient. A constant predictor that always outputs the base rate is calibrated, but it has no discrimination.

A model can be useful in two different ways, and calibration addresses only one of them. Calibration asks whether the probability scale matches event rates: among cases assigned 0.8, is the positive-event rate about 80 percent? Discrimination asks whether the model separates positives from negatives: do positives tend to receive higher scores than negatives? AUC measures ranking:

Here $X^{+}$ and $X^{-}$ are independent draws from the positive and negative populations, both of which must occur, and $s$ is the score. AUC gives half credit to a tie. It depends only on score order and ties, so any strictly increasing transformation of the scores leaves AUC unchanged. A model can have high AUC and poor calibration. For example, it might assign scores near 0.99 to positives and near 0.98 to negatives. The ranking is nearly perfect, but the probabilities are not trustworthy. Conversely, the constant population base-rate forecast is calibrated and has AUC equal to 0.5 when both classes occur.

If we want a single number to reward honest probabilities, we need a loss whose best possible report is the truth. That is the idea of a proper scoring rule. For a binary outcome with true probability $p$, a scoring rule $S(q,y)$ is proper if the following inequality holds for every reported probability $q$:

where:

• $S(q,Y)$: the score assigned to reported probability $q$ under outcome $Y$
• $p$: the true probability that $Y=1$
• $q$: the reported probability we are evaluating
• $\mathbb{E}_{Y\sim \mathrm{Bernoulli}(p)}[\cdot]$: the expectation over outcomes drawn from a Bernoulli distribution with success probability $p$

For a strictly proper rule, equality holds only when $q=p$. The Brier score is the mean squared error of the probabilities:

where:

• $n$: the number of cases
• $p_i$: the predicted probability for case $i$
• $y_i$: the observed outcome for case $i$, coded as 1 for positive and 0 for negative

To see why it is proper, fix a true probability $p$ and suppose we report $q$. The expected squared error is

where:

• $q$: the reported probability
• $Y$: the outcome, which is 1 with probability $p$ and 0 with probability $1-p$
• $p$: the true probability that $Y=1$

The second term does not depend on $q$, and the first term is minimized at $q=p$. Log loss is the negative log likelihood:

For a true probability $p$ and reported $q$, the expected log loss is

where:

• $p$: the true probability that $Y=1$
• $q$: the reported probability

For $0<p<1$ and $0<q<1$, differentiating with respect to $q$ gives

The minimum is unique because the second derivative is $p/q^2+(1-p)/(1-q)^2>0$ in the interior. For $p=0$ or $p=1$, the minimum occurs at the matching endpoint, using the usual $0\log0=0$ limit. Log loss therefore has the same properness property. It penalizes confident mistakes much more strongly than the Brier score because the logarithm is unbounded near zero. Thresholded accuracy cannot identify the true probability: many reports on the same side of the threshold give the same label. At threshold one half, the corresponding zero-one loss is weakly proper under the definition above, but it is not strictly proper. AUC is a pairwise ranking measure rather than a per-observation Bernoulli loss $S(q,y)$, and it does not identify the numerical probability scale.

The properness condition has a visual signature: for a fixed true probability, the expected Brier score is smallest when the reported probability equals the truth. The next figure draws that expected loss for three true probabilities and marks each minimum.

The formulas above describe an ideal conditional frequency. In practice we estimate that frequency by grouping predictions into bins and comparing the average prediction in each bin with the observed positive rate. A reliability diagram is the graphical version of that comparison, and it is often the first diagnostic we inspect. Suppose bin $b$ contains $n_b$ cases, with mean predicted probability

and observed positive frequency

where:

• $b$: the index of the reliability bin
• $n_b$: the number of cases in bin $b$
• $p_i$: the predicted probability for case $i$
• $y_i$: the observed outcome for case $i$
• $\mathrm{conf}_b$: the mean predicted positive-event probability in bin $b$
• $\mathrm{acc}_b$: the observed positive rate in bin $b$, rather than thresholded classification accuracy

We plot $\mathrm{conf}_b$ on the horizontal axis and $\mathrm{acc}_b$ on the vertical axis. The diagonal is perfect calibration. A point below the diagonal means the model overpredicts the positive-event rate; a point above means it underpredicts that rate. Calling either direction overconfidence across the whole binary probability scale would confuse positive-event risk with confidence in the more likely class. The number of bins and the binning strategy matter: equal-width bins can put very few observations in extreme bins, while equal-frequency bins can hide poor calibration near the extremes. A reliability diagram is a diagnostic, not a single-number score.

The expected calibration error summarizes the same binned comparison:

where:

• $B$: the number of bins
• $n_b$: the number of cases in bin $b$
• $n$: the total number of cases
• $\mathrm{acc}_b$: the observed positive rate in bin $b$
• $\mathrm{conf}_b$: the mean predicted probability in bin $b$

ECE depends on the binning scheme. Merging bins can cancel oppositely signed gaps and reduce ECE, even when the individual forecasts are unchanged. Tied scores can also make quantile-bin counts unequal or reduce the number of distinct bins. It should not be used as the sole calibration metric. For binned forecasts, where each case in bin $b$ is represented by the bin mean $\mathrm{conf}_b$, the Brier score decomposes as

where:

• $\mathrm{BS}_{\text{bin}}$: the Brier score of the binned forecasts
• $\mathrm{REL}$: the reliability term, the calibration penalty
• $\mathrm{RES}$: the resolution term, the discrimination reward
• $\mathrm{UNC}$: the score of the empirical constant base-rate forecast

and the reliability term is

the resolution term is

where:

• $n_b$: the number of cases in bin $b$
• $n$: the total number of cases
• $\mathrm{acc}_b$: the observed positive rate in bin $b$
• $\bar y$: the overall base rate, the fraction of positive cases

and the uncertainty term is

This identity is exact for the forecasts after every score in a bin is replaced by that bin's mean. It need not equal the Brier score of the original varying scores. Reliability is a penalty: it is large when predicted and observed frequencies disagree. Resolution is large when bins have event rates far from the overall rate. It measures this bin-level refinement, rather than ROC-AUC. Uncertainty is the Brier score of the empirical constant base-rate forecast on these observations. A useful summary is the Brier skill score relative to the base-rate forecast:

where:

• $\mathrm{BSS}$: the Brier skill score relative to the base-rate forecast
• $\mathrm{BS}$: the Brier score of the model being evaluated
• $\mathrm{BS}_{\text{ref}}$: the Brier score of the reference base-rate forecast, required to be positive

BSS is undefined when the reference score is zero. A reference using the evaluation sample's own base rate is a descriptive benchmark; a deployable constant forecast must be estimated before final evaluation.

Once the reliability diagram shows miscalibration, the next question is how to repair the probability scale without discarding the model. The repair is a mapping from raw score to calibrated probability, learned on held-out data. Sigmoid recalibration, often called Platt scaling, fits a two-parameter monotone map from the raw score to a calibrated probability. When the raw model outputs a probability $0<p_i<1$, we can work on its logit scale:

where:

• $p_i$: the raw probability output by the base model for case $i$
• $z_i$: the logit of that probability, which is the input scale for sigmoid recalibration
• $\operatorname{logit}(\cdot)$: the log-odds transformation

The calibrated probability is

where:

• $q_i$: the calibrated probability for case $i$
• $z_i$: the logit of the raw model probability
• $a$: the slope parameter on the logit scale
• $b$: the intercept parameter on the logit scale
• $\sigma(\cdot)$: the logistic sigmoid function

The parameters $a$ and $b$ are fitted on the calibration set by minimizing log loss:

where:

• $\ell(a,b)$: the negative log likelihood on the calibration set as a function of the slope and intercept
• $n$: the number of calibration cases
• $q_i$: the calibrated probability for case $i$, equal to $\sigma(a z_i+b)$
• $y_i$: the observed outcome for case $i$

The derivative of the logistic sigmoid satisfies $\sigma'(\eta)=\sigma(\eta)(1-\sigma(\eta))$, and the derivative of $\log\sigma(\eta)$ is $1-\sigma(\eta)$. Applying the chain rule gives

where:

• $\ell$: the negative log likelihood from above
• $q_i$: the calibrated probability for case $i$
• $y_i$: the observed outcome for case $i$
• $z_i$: the logit of the raw model probability for case $i$

Setting these derivatives to zero gives the first-order conditions. The general first-order system has no direct closed-form solution, so we use a numerical optimizer such as BFGS. With separated calibration labels, an unregularized finite optimum need not exist. When $a=1$ and $b=0$, the map is the identity. With $b=0$ and $0<a<1$, the map moves probabilities toward one half; with $b=0$ and $a>1$, it moves them toward the endpoints. A nonzero intercept shifts the curve. Positive $a$ preserves order, negative $a$ reverses it, and $a=0$ gives a constant. Our unconstrained fit must therefore be checked before invoking an order-preservation claim. If the raw score is a margin rather than a probability, as with a support vector machine, the same idea applies directly as $q=\sigma(a s+b)$.

Isotonic recalibration is nonparametric. It seeks a nondecreasing function $g$ that minimizes

where:

• $g(\cdot)$: the fitted nondecreasing mapping from raw score to calibrated probability
• $g(p_i)$: the fitted calibrated probability for case $i$
• $p_i$: the raw score for case $i$
• $y_i$: the observed outcome for case $i$
• $n$: the number of calibration cases
• $p_i\le p_j \Longrightarrow g(p_i)\le g(p_j)$: the monotonicity constraint ensuring a higher raw score does not produce a lower calibrated probability

The monotonicity constraint encodes the reasonable requirement that a higher raw score should not produce a lower calibrated probability. The pool adjacent violators algorithm solves this problem. We first aggregate equal scores so they receive the same fitted value, sort the distinct scores, and merge adjacent weighted blocks whenever their means violate monotonicity. The fitted value for each block is its weighted mean outcome. PAVA gives block-constant fitted values at the observed scores. Our predictor linearly interpolates between those score thresholds; any monotone extension matching the fitted points has the same training objective. Isotonic regression is flexible and can correct complex miscalibration, but it requires more calibration data and can overfit. Outside the range of observed scores it must extrapolate, typically by holding the boundary value constant.

Both methods are post hoc. They use the calibration set to learn a mapping and then apply that mapping to new scores. They leave the base model fixed and use only its score. The mapping can alter ties or score orientation, but supplies no new feature information.

A calibrated model satisfies $\mathbb{E}[P]=\Pr(Y=1)$. This follows from the law of total expectation. Calibration is not preserved by arbitrary transformations: if $P$ is calibrated and we apply a nonlinear function $h$, the transformed value $h(P)$ need not be calibrated. Discrimination is different: a strictly increasing transformation preserves both ranking and existing ties, hence AUC. Finite-precision saturation or clipping can introduce extra ties.

Proper scoring rules assess the numerical probability forecast. A model can improve its Brier score by becoming better calibrated, by becoming better at separating classes, or both. For the constant population forecast $\pi=\Pr(Y=1)$, the expected Brier score is $\pi(1-\pi)$. A finite-sample score need not equal this expectation. A model with strong discrimination but poor calibration can have a worse Brier score than a calibrated model with weaker discrimination. This is why we report calibration diagnostics and proper scores together rather than relying on a single number.

A small numerical example makes the decomposition concrete and shows why calibration can improve proper scores without changing the ranking. Suppose a model produces ten predictions and we group them into two reliability bins. Every case in the high-score bin receives probability 0.9; four of its five cases are positive. Every case in the low-score bin receives probability 0.6; one of its five cases is positive. Using only bin means would not determine the original unbinned scores. We fit and score the illustrative mapping on these same ten cases to show the arithmetic; the later implementation uses separate calibration and test partitions.

The raw Brier score is the average squared difference between each predicted probability and its outcome.

In the low bin, one case has outcome 1 and four have outcome 0:

The overall Brier score is

Now we decompose it. The reliability term is

The resolution term is

The uncertainty term is

where $\bar y=0.5$ is the base rate, the fraction of positive cases in the full sample. So

which matches the direct Brier score. The expected calibration error is

The raw log loss is

Now apply an isotonic recalibration. The high bin maps to 0.8 and the low bin maps to 0.2, because those are the observed rates in the two monotone blocks. The calibrated Brier score becomes

The calibrated log loss becomes

Both proper scores improve substantially. The ranking, however, does not improve. There are four positive cases in the high bin and one positive case in the low bin, with one negative case in the high bin and four negative cases in the low bin. Pairing each positive with each negative gives 25 pairs. Predictions are tied within each bin both before and after this mapping. There are sixteen cross-bin pairs with the positive in the high bin and the negative in the low bin, four within-high-bin tied pairs, four within-low-bin tied pairs, and one cross-bin pair with the positive in the low bin and the negative in the high bin. Treating ties as one-half gives

The calibrated scores preserve this ranking, so AUC remains 0.8. This example shows the central distinction: calibration repairs the probability scale, while discrimination is about the ordering. That is why we inspect both calibration and ranking.

The additive decomposition can be read as a signed account: reliability adds penalty, resolution subtracts a reward, and uncertainty is the empirical constant base-rate term. The next figure shows those terms for the two-bin example and checks that they add to the raw Brier score.

We now build the calibration workflow from primitives and compare it with standard library implementations. The implementation follows the logic of the formulas: fit the base classifier on training data, fit each calibrator on a separate calibration partition, and evaluate the results on an untouched test partition. We fit a Gaussian naive Bayes classifier. To make a controlled scale error visible, we then multiply its predicted log odds by two and convert them back to probabilities. This deliberately sharpens the scores without adding feature information; it is a synthetic demonstration, not a claim that this classifier is always overconfident.

The next cells import the libraries and set the data-generation seed, then generate and split the data. The calibration and test predictions use rows held out from base-model fitting.

Below are the main parameters that affect how the calibrators work and how reliable the resulting probabilities are.

Calibration-set size and support across the score range are important practical constraints. Sigmoid recalibration has only two parameters and can work with a relatively modest calibration sample, but isotonic recalibration can fit many blocks and needs more data to avoid overfitting. There is no universal threshold that works for every problem; the relevant question is whether the calibration set contains enough positives and negatives in the score regions that matter for decisions.

The choice is a flexibility tradeoff. Sigmoid fits a smooth two-parameter curve, which can miss a more irregular calibration relationship. Isotonic fits a nondecreasing relationship with plateaus that can create ties; it can overfit a small calibration sample. Check support across the score range and compare methods using validation data that remains separate from final evaluation.

The reliability-diagram parameters are the number of bins and the binning strategy. Quantile bins put roughly the same number of observations in each bin, which stabilizes the observed rates but can widen bins in sparse probability regions. Uniform bins keep the probability axis evenly spaced but can produce noisy extreme bins. The ECE depends on both choices, so we report it alongside Brier score and log loss rather than in isolation.

EPS clips only logits and log-loss evaluation, so the computed log loss is that of the clipped forecast. The ideal log loss can be infinite at a confidently wrong endpoint. Brier scoring uses the original validated probabilities. SCORE_SHARPNESS is fixed at two for this controlled example; it is not selected using the test labels.

The following are the main functions we use for calibrating and evaluating probabilities.

• fit_sigmoid and predict_sigmoid implement Platt scaling on the logit scale.

• fit_isotonic and predict_isotonic implement the pool adjacent violators algorithm and monotone interpolation.

• brier_score computes the exact binary squared-error score, while log_loss_score evaluates the numerically clipped forecast on the test set.

• reliability_table bins predictions for reliability diagrams.

• expected_calibration_error summarizes the binned gaps.

• roc_auc_score measures discrimination and is used to verify that calibration has not been mistaken for a ranking improvement.

The reliability diagram is the main diagnostic for calibration. We compute it on the untouched test set. The deliberately sharpened raw scores illustrate scale miscalibration. The sigmoid map estimates a smooth correction, but its curve on the separate test partition need not lie on the diagonal. Isotonic recalibration can bend more sharply, but it can also create flat segments where many scores receive the same calibrated probability.

Calibration is warranted when the probability itself is an input to a decision, rather than only a device for ordering cases. Expected-cost thresholds, risk communication, and monitoring predicted event rates use the number on the probability scale. A capacity-constrained queue may instead need only ranking when it selects a fixed number of otherwise comparable cases. When review decisions depend on case-specific expected costs or benefits, probability values matter as well. In a fraud queue, for example, an expected-cost rule needs a score that can be multiplied by review cost and missed-loss; in credit or insurance, predicted default or claim rates are compared with realized rates across segments; in clinical settings, a risk score may be shown to patients or used to allocate follow-up care. When the only requirement is to rank cases from highest to lowest risk, calibration is optional and AUC or another ranking measure is the more direct diagnostic.

A useful practice is to report calibration and discrimination together. The reliability diagram shows where the probability scale fails, Brier score and log loss summarize the size of the failure in proper scoring terms, and AUC shows whether the ordering remains useful. The expected calibration error can be included for communication, but it is bin-dependent and should not be the only calibration number. A model that improves Brier score after recalibration has not necessarily improved its ranking; strictly increasing maps preserve ranking, while isotonic ties or a negative sigmoid slope can change it.

To fit a calibrator, use a calibration partition that is separate from both training and final test data. When labeled data are scarce, cross-fitting can generate out-of-fold scores for calibration, but the final evaluation must still use labels that were not used to fit the base model or the calibrator. For modest calibration sets, a two-parameter sigmoid is a lower-flexibility starting point when both classes and the relevant score range are represented. Its predictions outside that range still need validation. Isotonic recalibration becomes more attractive when the calibration set is large enough to support stable block estimates and the reliability diagram shows a pattern that a logistic map cannot capture. A reasonable diagnostic starting point is ten quantile bins, repeated with five and twenty bins to reveal sensitivity; inspect the number of observations and events per bin before interpreting a gap.

Computational cost depends on the workload. Sigmoid calibration optimizes two parameters on a one-dimensional score, so each objective evaluation is linear in the calibration-set size. Isotonic calibration requires sorting the scores and running the pool adjacent violators algorithm, which is dominated by the sort and uses linear memory. Proper scores cost $O(n)$. After computing bin edges, this teaching reliability loop scans the observations once per bin, costing $O(nB)$ for $B$ bins; quantile-edge construction adds order-statistic work. For a fixed small bin count it scales linearly with $n$. For very large datasets, the calibration sample can be capped or stratified, provided that the score regions used in decisions still contain enough positives and negatives. In deployment, a sigmoid calibrator is stored as two numbers, while an isotonic calibrator is stored as a set of thresholds and fitted values; sigmoid evaluation is constant work per case, while isotonic prediction searches the stored thresholds.

Several important failures are procedural. Fitting a calibrator on training predictions and then evaluating on those same cases measures in-sample fit. It does not show that calibration has corrected base-model overfitting or that the mapping will generalize. Evaluating repeatedly on the final test set while choosing among calibrators also leaks information and makes the reported score optimistic. Another frequent mistake is treating calibration as a substitute for discrimination: a monotone map cannot separate classes that the base model already ranks together. Isotonic calibration can overfit a small calibration set, create flat regions, and produce ties, while any calibrator must extrapolate outside the observed score range; isotonic methods typically hold the boundary value constant. Finally, ECE alone can be reduced by coarse binning or by avoiding extreme probabilities, so it should be read alongside proper scores and the reliability diagram.

The calibration set must resemble the population where the probabilities will be used. Keep the label definition and preprocessing consistent, and cover the score-outcome relationships expected in deployment; a calibrator learned on a convenient sample can be systematically wrong on the target population. Check the positive and negative counts in the score regions used for decisions. No fixed sample count guarantees a stable sigmoid fit. Both score spread and positive/negative counts matter, while the flexible isotonic fit generally needs more information across the score range. Reporting bootstrap intervals for bin-level observed rates, or repeating the calibration split, helps distinguish a real calibration gap from sampling noise.

Aggregate calibration can hide subgroup failures. A model may be calibrated overall while being overconfident for one age band, region, product, or facility. When subgroup sample sizes permit, compute reliability diagrams and proper scores within the subgroups that matter for deployment, and inspect whether the gaps are stable rather than isolated to sparse bins. If a subgroup is large enough and the gap persists, options include group-specific calibration, adding the subgroup variable to the calibration model, or collecting more representative data. These choices affect fairness and governance, so they should be documented rather than absorbed into an aggregate metric.

Distribution shift is the main reason calibration should be treated as temporary evidence. Under covariate shift, the input distribution changes while the outcome distribution conditional on the inputs stays the same. Under label shift, the class base rate changes while the input distribution conditional on the class stays the same. Concept shift changes the relationship between inputs and outcomes. Any of these can invalidate a previously fitted map. If labels from the target population are available, use them to check calibration and refit the mapping when that evidence supports an update. If only the base rate is known to have changed, a prior-shift correction may help, but it does not guarantee that the full reliability curve is correct. After deployment, monitor reliability diagrams, Brier score, log loss, AUC, and subgroup gaps, and version the calibrator alongside the base model. Reassess calibration when the population or label definition changes, and revalidate the decision rule when costs or decision context change. Recalibrate when evidence from the target population shows that the relationship between scores and outcomes needs updating. Do not claim that calibration remains valid after arbitrary shift without evidence from the shifted population.

Probability calibration makes predicted probabilities match observed frequencies. It is separate from discrimination: a model can rank cases well while being overconfident, and a model can be calibrated while having no ranking power. Reliability diagrams inspect calibration, while Brier score and log loss evaluate overall probability quality. Brier score and log loss are proper scoring rules, meaning they are minimized in expectation by the true probability. AUC measures ranking and is not a calibration metric. The expected calibration error is a useful diagnostic, but it depends on binning and is not proper.

Sigmoid recalibration fits a two-parameter logistic map on the logit scale; its limited flexibility can help with smaller samples, but estimation still needs adequate class and score support. Isotonic recalibration uses the pool adjacent violators algorithm for monotone block estimates and interpolates between score thresholds for prediction. Both methods are post hoc, need separate calibration data, and consume only the existing model score. Strictly increasing recalibration preserves ranking. Ties and negative fitted slopes need separate checks; calibration should be evaluated for probability reliability rather than promoted as a ranking improvement.

The practical workflow is to train the base model, calibrate on a separate representative partition, and evaluate on untouched test data with reliability diagrams, Brier score, log loss, AUC, and subgroup checks. Calibration is a property of the model-data pair, not a permanent guarantee. It should be monitored over time and reassessed after population or base-rate changes. Re-estimate the probability map when evidence from the target population supports an update, and revalidate the decision rule when costs or decision context change.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about probability calibration.