Calibration › The Calibration Problem

To understand calibration concretely, consider a binary classification scenario. Let $\hat{Y}$ be the predicted label and $\hat{P}$ be the predicted probability (confidence) for that label. A model is perfectly calibrated if, for any confidence value $p \in [0, 1]$:

$$P(\hat{Y} = Y \mid \hat{P} = p) = p$$

where:

• $\hat{Y}$: the predicted label
• $Y$: the true label
• $\hat{P}$: the predicted probability (confidence) for the predicted label
• $p$: a specific confidence value in $[0, 1]$

This definition captures the intuitive notion that predictions carrying a specific confidence level should materialize with exactly that frequency. Mathematically, it states that the conditional probability of correctness given a confidence score must equal the confidence score itself. This means that among all predictions where the model claims probability $p$, the fraction that are correct should equal $p$. If the model outputs confidence 0.9 for a thousand different predictions, exactly nine hundred of them should be correct.

The definition looks simple, but achieving it during training is difficult. Notice that the formula says nothing about individual predictions: it is a statement about the statistical behavior of the model over many predictions at each confidence level. A single prediction with confidence 0.9 can be either right or wrong without violating calibration. What matters is the long-run empirical frequency. This is fundamentally a frequentist notion of probability: the model's stated probabilities must match empirical frequencies when aggregated across many predictions with similar confidence values.

In practice, we work with multiclass classification where language models output a distribution over the vocabulary. For a generated token or classification label, we typically consider the maximum probability (top-1 confidence) or the probability assigned to the specific target class. The extension from binary to multiclass settings requires careful consideration of which probability mass to evaluate.

The three most common formulations of multiclass calibration are:

• Top-1 (confidence) calibration: The probability assigned to the predicted class (the argmax class) should match how often that prediction is correct. This is the most natural extension of binary calibration and the one most commonly measured in practice.
• Classwise calibration: Each class's probability distribution should be calibrated independently. If a model assigns probability 0.7 to class "cat" across many predictions, cat should appear 70% of the time among those predictions. This is a stronger requirement than top-1 calibration because it demands calibration for every class simultaneously.
• Canonical calibration: The full predicted probability vector should satisfy a multi-dimensional calibration condition. This is the strongest requirement and the hardest to achieve or measure.

For most practical purposes, top-1 calibration is what practitioners care about because it governs decision thresholds for acceptance and rejection. When you configure an automated system to "act on predictions above 0.8 confidence," you need top-1 calibration to be accurate. If you care about the full distribution for reasons such as ranking multiple candidates or computing entropy-based uncertainty, classwise calibration becomes more important.

The modern deep learning training regime systematically produces overconfident models. This might seem counterintuitive: training on cross-entropy loss directly involves probabilities, so why would the learned probabilities be wrong?

The answer lies in the asymmetry of the cross-entropy gradient. When the model assigns probability $p$ to the correct class, the gradient of cross-entropy with respect to the logit for that class is $-(1 - p)$. This gradient is large when $p$ is small (the model is uncertain) and shrinks toward zero as $p$ approaches 1.0. The gradient is zero at perfect confidence. This means the training signal always pushes the model toward higher confidence in correct answers, and the signal becomes weaker as confidence increases. The optimization naturally gravitates toward high-confidence predictions because that is where the loss approaches zero. There is no corresponding force that prevents confidence from being calibrated; the loss only cares about discrimination, not about whether the confidence levels are meaningful.

Early neural network literature noted calibration as a property that networks should naturally exhibit, but empirical observations eventually contradicted this optimism. Guo et al. (2017) famously documented that modern deep neural networks, despite being far more accurate than their predecessors, are significantly less calibrated. They found that a ResNet trained on CIFAR-100 achieved 70% accuracy with only modest calibration error in 1990s-era networks but severe overconfidence in modern deep residual networks. The culprit was a combination of increased model capacity, batch normalization, and weight decay applied without calibration-awareness. Each advance that improved accuracy tended to worsen calibration.

For large language models, the calibration picture is more complex because the models produce probabilities over thousands of vocabulary tokens at each position, and the relationship between token-level probabilities and downstream task accuracy is indirect. A model's confidence in its next token does not directly correspond to its confidence in a complete multi-step answer. This gap between token-level calibration and answer-level calibration is an active area of research and has implications for how we should interpret language model confidence estimates in practice.