Calibration › Calibration Methods

When a model is poorly calibrated, we can apply post-hoc calibration methods that adjust the output probabilities without retraining the model or changing its architecture. These methods treat the trained model as a fixed feature extractor and learn a transformation that maps uncalibrated probabilities to calibrated ones. This approach is computationally efficient because it requires only a small validation set to learn the calibration parameters, avoiding the expense of retraining the entire model.

The standard workflow for post-hoc calibration is:

1. Train the model normally on the training set
2. Collect predictions (logits or probabilities) on a held-out calibration set
3. Learn the calibration mapping on the calibration set using maximum likelihood or mean squared error
4. Apply the learned mapping to all future predictions

The calibration set should be large enough to reliably estimate calibration parameters and diverse enough to cover the confidence range you care about. Using the validation set for both hyperparameter selection and calibration is a common practice, but it risks slight overfitting to the validation distribution. For the best practice, maintain a separate calibration set that was never used for any other purpose during model development.

Temperature scaling is the simplest and most widely used calibration method for neural networks. It works by dividing the logits (pre-softmax scores) by a scalar temperature parameter $T > 0$ before applying the softmax function. This scaling operation softens or sharpens the probability distribution without changing the relative ordering of the logits, meaning the model's predictions (which class it chooses) remain unchanged while only the confidence values are adjusted.

Given logits $\mathbf{z}$ for each class, the calibrated probability for class $i$ becomes:

$$p_i = \frac{\exp(z_i / T)}{\sum_{j} \exp(z_j / T)}$$

where:

• $p_i$: the calibrated probability for class $i$
• $z_i$: the logit (pre-softmax score) for class $i$
• $T$: the temperature parameter ($T > 0$)
• $j$: index iterating over all classes

When $T = 1$, we recover the original model. When $T > 1$, the probability distribution becomes softer, meaning more uniform, reducing confidence. When $T < 1$, the distribution becomes sharper, increasing confidence. Geometrically, increasing the temperature pulls the probabilities toward the uniform distribution $1/K$ for $K$ classes, while decreasing it pushes them toward one-hot vectors.

The intuition behind temperature scaling connects to the physics of thermodynamics, where the term "temperature" originates. In a physical system, higher temperature means more random thermal fluctuations, leading to more uniform distributions over states. Lower temperature means the system is more likely to settle into low-energy states, creating a peaked distribution. The Boltzmann distribution $p_i \propto \exp(-E_i / T)$ is exactly the softmax applied to negative energies scaled by temperature. Neural network logits play the role of negative energies, and temperature scaling applies the same principle to probability distributions over classes.

The optimal temperature $T$ is learned on a validation set by minimizing the negative log-likelihood (NLL). This optimization is univariate (only one parameter to fit), has a unique minimum under mild conditions, and can be solved efficiently with any scalar optimization method. Since temperature scaling preserves the ranking of logits, it does not change the model's predictions (argmax), only the confidence values. This makes it ideal for calibration without sacrificing accuracy.

Temperature scaling works remarkably well for many deep learning models, particularly those using cross-entropy loss, because it addresses the general overconfidence problem with a single parameter. The universality of overconfidence in neural networks means that a single scalar adjustment often suffices to significantly improve calibration across the entire confidence range. Guo et al. (2017) found that temperature scaling achieved competitive or superior calibration performance compared to more complex methods on many standard benchmarks, an impressive result given its simplicity.

The limitation of temperature scaling is its single degree of freedom. It assumes that miscalibration is uniform: the same temperature corrects the model's confidence at all confidence levels. If the model is overconfident at high confidence but underconfident at medium confidence, temperature scaling cannot fix both problems simultaneously. In practice, this limitation is minor for many models because overconfidence tends to be the dominant direction across the full confidence range.

Platt scaling, originally developed for Support Vector Machines by John Platt in 1999, fits a logistic regression model to the classifier outputs. Where temperature scaling uses a single parameter to rescale all logits uniformly, Platt scaling uses two parameters: a slope and a bias. This two-parameter affine transformation can correct for both the scale of logits (like temperature scaling) and a systematic offset in the logit values. For binary classification:

$$p(y=1 \mid z) = \sigma(az + b) = \frac{1}{1 + \exp(-(az + b))}$$

where:

• $p(y=1 \mid z)$: the calibrated probability of the positive class given logit $z$
• $z$: the model's logit (or confidence score) for the positive class
• $\sigma(\cdot)$: the sigmoid function
• $a$: the slope parameter (learned on the calibration set)
• $b$: the bias parameter (learned on the calibration set)

The bias parameter $b$ allows Platt scaling to correct for models that are systematically biased toward one class, a situation that pure temperature scaling cannot address because it preserves the relative ordering and marginal distribution of predictions. For multiclass problems, Platt scaling can be applied to each class independently in a one-vs-rest fashion, fitting a separate $(a_k, b_k)$ pair for each class $k$.

Unlike temperature scaling, Platt scaling can change the ranking of predictions because the affine transformation with a non-unit slope and non-zero bias does not preserve all orderings. It is more flexible than temperature scaling but requires more parameters and is therefore more prone to overfitting on small validation sets. The additional degrees of freedom allow Platt scaling to correct more complex miscalibration patterns, but this flexibility requires careful regularization or larger calibration sets to avoid learning noise in the calibration data.

When should you prefer Platt scaling over temperature scaling? If you have evidence that the model's logits have a systematic bias, such as always predicting too confidently in the positive direction for one class, Platt scaling can address this through its bias term. If the miscalibration is purely a matter of the logit scale being off (the model ranks examples correctly but assigns unreliable confidence scores), temperature scaling is simpler and less prone to overfitting.

Isotonic regression is a non-parametric calibration method that learns a piecewise constant, monotonically increasing function mapping uncalibrated probabilities to calibrated ones. This approach makes no assumptions about the functional form of the calibration error beyond the reasonable constraint that higher uncalibrated confidence should map to higher calibrated probability. It finds the function $f$ that minimizes the mean squared error between calibrated outputs and true outcomes:

$$\min_{f} \sum_{i=1}^{n} (f(\hat{p}_i) - y_i)^2$$

where:

• $f$: the isotonic (non-decreasing) calibration function being learned
• $\hat{p}_i$: the uncalibrated predicted probability for the $i$-th sample
• $y_i$: the true binary outcome (0 or 1) for the $i$-th sample
• $n$: the number of validation samples

subject to the constraint that $f$ is isotonic (non-decreasing): $f(\hat{p}_i) \leq f(\hat{p}_j)$ whenever $\hat{p}_i \leq \hat{p}_j$.

The monotonicity constraint is important and reasonable: if the model assigns higher confidence to one prediction than another, the calibrated probability should also be at least as high. This preserves the relative ordering of predictions, which is the fundamental information we want to keep while correcting the absolute probability values.

The isotonic regression problem is solved by the pool adjacent violators algorithm (PAVA), a linear-time algorithm that merges adjacent bins whenever the monotonicity constraint is violated. Starting from the sorted list of predictions, PAVA identifies pairs of adjacent predictions that violate the monotone constraint (where the fitted value decreases) and pools them by averaging. This pooling continues until no violations remain.

Isotonic regression makes no assumptions about the shape of the calibration error beyond monotonicity, making it more flexible than parametric methods. However, it requires more data to avoid overfitting and does not generalize well beyond the range of confidence values observed during calibration. If the validation set contains no examples with confidence below 0.2, the isotonic regression function may behave unpredictably when encountering such low-confidence predictions at test time. It is particularly useful when the calibration curve has a complex, non-linear shape that temperature scaling cannot capture, such as when the model is underconfident at low probabilities but overconfident at high probabilities.

Beta calibration extends Platt scaling by using the Beta distribution's cumulative distribution function, which is more naturally suited to probabilities bounded in $[0, 1]$. Standard logistic regression assumes logits can range over the entire real line, but probabilities naturally live in a bounded interval. The Beta distribution respects these natural bounds and can model asymmetric calibration curves more effectively.

Beta calibration applies a parametric transformation to the uncalibrated probability $\hat{p}$:

$$\log \frac{p}{1-p} = a \cdot \log \hat{p} - b \cdot \log(1 - \hat{p}) + c$$

where:

• $p$: the calibrated probability
• $\hat{p}$: the uncalibrated probability
• $a, b$: shape parameters controlling the steepness of calibration correction for low and high probabilities respectively
• $c$: an intercept term

The parameters $a$, $b$, and $c$ are learned by maximizing likelihood on the calibration set. The Beta calibration model reduces to Platt scaling when $a = b$, making it a strict generalization. The asymmetry between $a$ and $b$ is the key advantage: the model can apply stronger correction at high confidence levels without distorting low confidence levels, or vice versa.

This approach respects the natural bounds of probability values and can handle asymmetric calibration curves better than standard Platt scaling. When the calibration error differs between low and high confidence regions, Beta calibration can capture these nuances through the flexibility of its separate shape parameters for each end of the probability spectrum.

For models where different classes have different calibration profiles, class-independent temperature scaling is insufficient. Vector scaling generalizes temperature scaling by learning a separate temperature for each class, applying an element-wise multiplication of logits by a learned vector:

$$\mathbf{p} = \text{softmax}(\mathbf{W} \odot \mathbf{z})$$

where $\mathbf{W}$ is a diagonal matrix of per-class temperature parameters and $\odot$ denotes element-wise multiplication. This allows the model to be made more confident about some classes and less confident about others based on where the calibration errors lie.

Matrix scaling takes this further by learning a full $K \times K$ matrix transformation of the logits. This is the most expressive parametric post-hoc calibration method but also the most prone to overfitting when the calibration set is small relative to the number of classes. For models with thousands of vocabulary tokens, matrix scaling is impractical: a full matrix for a 50,000-vocabulary model would require $50{,}000^2 = 2.5$ billion parameters. Vector scaling (with $K$ parameters) is more tractable but still requires a calibration set with many examples per class.

While technically a training modification rather than post-hoc calibration, label smoothing deserves mention here because it directly addresses calibration during training. By replacing hard targets (0 and 1) with soft targets (e.g., 0.1 and 0.9), label smoothing prevents the model from becoming overconfident during training. This regularization technique encourages the model to maintain some uncertainty even when it correctly classifies training examples. A standard cross-entropy loss with label smoothing $\epsilon$ uses the modified target:

$$y_i^{\text{smooth}} = (1 - \epsilon) \cdot y_i + \frac{\epsilon}{K}$$

where $y_i$ is the original one-hot target, $\epsilon$ is the smoothing parameter (typically 0.1), and $K$ is the number of classes. The model is now rewarded for assigning at most $(1 - \epsilon + \epsilon/K)$ probability to the correct class, which prevents logits from growing without bound.

Label smoothing often results in better-calibrated models out of the box, with the trade-off of a potential small cost to accuracy. The optimal level of smoothing depends on model size, dataset size, and the specific task. Research on large language model calibration suggests that some amount of label smoothing is almost always beneficial for calibration while having minimal impact on accuracy when the smoothing parameter is in the range of 0.05 to 0.2.