Calibration › Calibration in Language Models

The calibration story for modern large language models is more complicated than for classification models, and it comes with unique challenges that standard calibration methods do not fully address. Understanding these challenges is important before applying calibration techniques to LLM outputs in practice.

A language model produces a probability distribution over the vocabulary at each token position. These token-level probabilities are well-defined and can be measured directly. However, the quantity we usually care about for calibration purposes is not token probability but answer probability: how confident is the model that its multi-token response is correct?

Converting from token probabilities to answer probabilities is non-trivial. One approach is to sum the log probabilities of all tokens in the response and use this as a log-probability for the full answer. This treats the model as having assigned the product of token probabilities to the full sequence. But this approach penalizes longer answers regardless of correctness, confounds fluency (high probability of generating grammatical tokens) with factual accuracy (generating the correct answer), and does not account for the many equivalent phrasings of the same correct answer.

Some evaluations compute the probability of a single correct token (such as "A", "B", "C", or "D" in a multiple-choice question) conditioned on the question and answer prefix. This approach isolates the relevant probability mass but only works for constrained answer formats. For open-ended questions, you typically need to either sample multiple completions and estimate confidence from the distribution, or use prompted techniques like asking the model to express its own confidence verbally.

One growing line of research asks language models to verbalize their own confidence. Instead of reading the model's internal probability scores, you prompt the model to say "I am 85% confident that..." and then evaluate whether that stated confidence is calibrated. This approach is appealing because it uses the model's language capabilities for uncertainty quantification, does not require access to logits (important for closed-weight models like commercial APIs), and can potentially capture epistemic uncertainty that token probabilities miss.

The research findings are mixed. Models can learn to express calibrated verbal confidence when explicitly trained on confidence calibration tasks, but they do not automatically do so from standard language modeling pre-training. When asked to express confidence without specific training for this skill, models often show systematic verbal overconfidence: they tend to use high-confidence language ("I am certain that...", "Definitely...") more often than their actual accuracy warrants. This verbal overconfidence is separate from the probability-level overconfidence and can be harder to correct because it reflects learned stylistic patterns in the training data rather than intrinsic probability estimates.

Large language models exhibit interesting scaling behavior with respect to calibration. Earlier research suggested that larger models tend to be more overconfident because they fit training data more aggressively. More recent work with models trained with explicit calibration objectives or RLHF finds that model scale does not uniformly worsen calibration: some large models are better calibrated than smaller ones when evaluated carefully on knowledge tasks.

The key factor appears to be what the model is confident about. Large models tend to be well-calibrated on questions they answer correctly most of the time (they correctly recognize that they know the answer) and poorly calibrated on questions where they make systematic errors (they confidently state incorrect facts). This "confident in wrong things" pattern is sometimes called hallucination confidence and represents a particularly dangerous form of miscalibration: the model does not just make errors, it expresses high confidence in those errors.

One practical response to calibration concerns is selective prediction: rather than always producing an answer, allow the model to abstain from answering when its confidence falls below a threshold. A well-calibrated model with an abstention option can achieve high accuracy on the questions it chooses to answer while avoiding the harmful effects of overconfident errors on difficult questions.

For selective prediction to work correctly, calibration is essential. If the model is overconfident, the threshold for abstention is harder to set correctly: you might accept answers that the model states with 85% confidence, but if that corresponds to 60% accuracy, your selective prediction policy is accepting too many incorrect answers. Conversely, if the model is underconfident, you might be causing unnecessary abstentions on questions the model would have answered correctly.

The coverage-accuracy trade-off is the key metric for selective prediction. Coverage is the fraction of queries the model answers (rather than abstaining), and accuracy is the fraction of answered queries that are correct. A perfectly calibrated model produces an optimal coverage-accuracy curve: by lowering the threshold, you can increase coverage at the cost of lower accuracy, with the trade-off following the empirical accuracy at each confidence level. A miscalibrated model produces a suboptimal curve: you get less accuracy at each coverage level than you would from a well-calibrated model.