Calibration › Sources of Miscalibration

Why do neural networks, including large language models, become miscalibrated? Several factors contribute, often interacting in complex ways during training and deployment. Understanding the sources helps you anticipate when calibration problems are likely and which remedies will be most effective.

Overfitting on the training objective. The cross-entropy loss (which we covered in Cross-Entropy Loss) optimizes for discrimination ability, not probability calibration. A model can minimize loss by being correct with high probability without learning the true likelihood of correctness. The training objective only penalizes the model for assigning low probability to the correct answer, but does not penalize assigning excessively high probability when the model is correct. This asymmetry encourages the model to push probabilities toward 1.0 whenever possible, regardless of the actual uncertainty. As a result, models that fit the training data well tend to be overconfident because the optimization pressure always pushes confidence upward on correct examples.

Model capacity and overconfidence. As models grow larger, they tend to become more overconfident. The increased capacity allows them to fit the training data more aggressively, pushing softmax probabilities toward 0 and 1 even when the true uncertainty remains high. Large models can memorize training examples or fit spurious correlations, leading to unjustified certainty in their predictions. This phenomenon appears paradoxical because larger models often generalize better in terms of accuracy, yet their confidence calibration degrades. The resolution is that generalization in terms of getting the right answer and generalization in terms of knowing how often you will get the right answer are different capabilities, and scale improvements thus far have primarily benefited accuracy over calibration.

Label smoothing effects. During training, label smoothing distributes some probability mass to incorrect classes, which can improve calibration by preventing the model from becoming too confident. A hard target of 1.0 for the correct class creates infinite gradient pressure to push confidence toward 1.0 indefinitely. A smoothed target of, say, 0.9 provides a finite ceiling: once the model reaches that confidence level on training data, the gradient approaches zero. This prevents the logits from growing without bound and forces the model to maintain some uncertainty. However, finding the optimal smoothing parameter requires careful tuning, and using too much smoothing can hurt both accuracy and the interpretability of confidence values.

Batch normalization interactions. Batch normalization, which rescales intermediate activations to have unit variance, can interact with calibration in unexpected ways. By controlling the scale of activations throughout the network, batch normalization changes how sensitive the final logit scale is to individual training examples. This can make the temperature of the softmax output layer effectively arbitrary: the logit values represent relative ranking more than absolute magnitude, meaning the raw softmax outputs may be systematically over- or under-confident in ways that are difficult to predict without post-hoc calibration.

Distribution shift. Models are often poorly calibrated on out-of-distribution data or domains different from their training distribution, even when they maintain reasonable accuracy. When the input distribution changes, the relationship between model confidence and true probability often breaks down. A model trained on news articles may be overconfident when processing medical texts, even if it sometimes produces correct answers. This compound challenge of maintaining both accuracy and calibration under distribution shift remains an active research area. Calibration under distribution shift is significantly harder than calibration within the training distribution because post-hoc calibration methods are typically learned on validation data from the same distribution as the training data.

RLHF and fine-tuning effects. Language models fine-tuned with reinforcement learning from human feedback (RLHF) can develop unusual calibration properties because the training signal rewards producing outputs that humans rate highly, not necessarily outputs where stated confidence matches accuracy. When a model has been trained to sound confident and authoritative because humans prefer assertive-sounding responses, the resulting confidence scores may be systematically inflated across all outputs regardless of actual accuracy. This creates a misalignment between the communicative function of confidence (reassuring the user) and the statistical function of calibration (informing accurate decision-making).