Hallucination Causes › Exposure Bias

One of the most important and underappreciated causes of hallucination is a mismatch between how the model is trained and how it generates text. This mismatch is called exposure bias, and it emerges from a fundamental tension between training efficiency and inference realism.

During training, language models use a technique called teacher forcing. At every training step, when computing the probability of the next token, the model conditions on the ground-truth previous tokens from the training data, not on its own previous predictions.

For an input sequence $y_1^*, y_2^*, \ldots, y_T^*$ where the asterisk denotes ground-truth tokens, training computes:

$$\mathcal{L}_{\text{train}} = \sum_{t=1}^{T} \log P\!\left(y_t^* \mid y_1^*, y_2^*, \ldots, y_{t-1}^*\right)$$

where:

• $y_t^*$: the ground-truth token at position $t$ (the correct answer the model should predict)
• $y_1^*, \ldots, y_{t-1}^*$: the ground-truth tokens at all prior positions (the "teacher" signal that forces correct context)
• $T$: the total sequence length
• $P(y_t^* \mid y_1^*, \ldots, y_{t-1}^*)$: the model's predicted probability for the correct token, conditioned on perfect context

The superscript $*$ throughout indicates that these are ground-truth tokens taken from training data, not tokens that the model generated itself.

Teacher forcing is computationally efficient. Because all positions in the sequence can be computed in a single forward pass using attention masking, training is fast and parallelizable. The gradient signal is also clean: at every step, the model is evaluated against the correct answer given the correct context, producing a well-defined and unambiguous learning signal.

But this efficiency comes at a cost. The training condition (always given perfect context) never occurs at inference time.

At inference time, the model has no access to ground-truth previous tokens. It must condition on its own previously generated tokens:

$$P\!\left(\hat{y}_t \mid \hat{y}_1, \hat{y}_2, \ldots, \hat{y}_{t-1}\right)$$

where:

• $\hat{y}_t$: the token the model generates at step $t$ (using a hat to distinguish from ground-truth $y_t^*$)
• $\hat{y}_1, \ldots, \hat{y}_{t-1}$: the model's own previously generated tokens, which form the inference context
• $P(\hat{y}_t \mid \hat{y}_1, \ldots, \hat{y}_{t-1})$: the model's predicted probability for its next token, conditioned on its own prior outputs

The key contrast: during training, the conditioning context is always $y_1^*, \ldots, y_{t-1}^*$ (perfect ground-truth). During inference, the conditioning context is $\hat{y}_1, \ldots, \hat{y}_{t-1}$ (the model's own outputs, which may contain errors).

The distributions $P(y_t^* \mid y_1^*, \ldots, y_{t-1}^*)$ and $P(\hat{y}_t \mid \hat{y}_1, \ldots, \hat{y}_{t-1})$ are fundamentally different, because the conditioning context differs. The model was never trained to recover from its own mistakes. If it generates a slightly off-topic or inaccurate token early in a sequence, it must condition all future tokens on that error. Training never prepared it for this scenario, and the model has no learned error-correction behavior to draw on.

The mathematical consequence is that the model's probability estimates are well-calibrated for contexts matching the training distribution (correct prefixes) but poorly calibrated for out-of-distribution contexts (error-containing prefixes). A model that is highly accurate when conditioned on perfect context can be substantially less accurate when conditioned on slightly perturbed context, because the perturbed context falls outside the space of inputs the model was trained on.

Exposure bias creates compounding errors. A small factual inaccuracy or semantic drift at position $t$ changes the conditioning context for position $t+1$. The model, never trained on such contexts, generates the statistically most likely continuation given that erroneous context, which may compound the original error rather than correct it.

This is particularly visible in long-form generation. Ask a model to write a multi-paragraph technical explanation or a detailed biography, and early factual errors tend to cascade: the model continues to build on its own (incorrect) premises rather than reversing course. Reversal would require generating tokens that conflict with the highly probable completions that the erroneous context suggests, and nothing in the model's training incentivizes such self-correction.

Suppose the model commits an error at step $t$. The log-probability of the correct answer at step $t+k$ given the error context is:

$$\log P\!\left(y_{t+k}^* \mid \hat{y}_1, \ldots, \hat{y}_{t-1}, \hat{y}_t, \ldots, \hat{y}_{t+k-1}\right)$$

where:

• $y_{t+k}^*$: the correct token $k$ steps after the error
• $\hat{y}_1, \ldots, \hat{y}_{t-1}$: correctly generated tokens before the error occurred
• $\hat{y}_t, \ldots, \hat{y}_{t+k-1}$: tokens generated after the error, each conditioned on the growing error context

Because the conditioning $\hat{y}_{1:t+k-1}$ diverges increasingly from the correct prefix $y_{1:t+k-1}^*$ as $k$ grows, the model encounters an increasingly out-of-distribution sequence of contexts. Its probability estimates for these contexts are unreliable by construction, and the error compounds with each step.

We can quantify this divergence using KL divergence between the model's behavior in the correct-context case and the error-context case:

$$\Delta_t = D_{\text{KL}}\!\left(P(y_{t+1} \mid y_{1:t}^*) \;\|\; P(y_{t+1} \mid \hat{y}_{1:t})\right)$$

where:

• $\Delta_t$: the divergence at step $t$ between the distribution conditioned on correct context and the distribution conditioned on error context
• $D_{\text{KL}}(P \| Q)$: the Kullback-Leibler divergence from distribution $Q$ to distribution $P$, measuring how much $P$ differs from $Q$
• $P(y_{t+1} \mid y_{1:t}^*)$: the model's next-token distribution conditioned on the correct prefix up to step $t$
• $P(y_{t+1} \mid \hat{y}_{1:t})$: the model's next-token distribution conditioned on the error-containing prefix up to step $t$

This divergence grows as the generated prefix diverges further from the true prefix. Since the model is never trained on diverged prefixes, $\Delta_t$ is not bounded by the training objective and can grow arbitrarily with sequence length. This is why you tend to observe more hallucinations in long responses than in short ones: the opportunities for error accumulation multiply with sequence length.

Exposure bias is not unique to language models. The same issue arises in any sequence-to-sequence system trained with teacher forcing, including neural machine translation systems from the pre-transformer era. Various approaches have been proposed to reduce it: scheduled sampling, which randomly replaces ground-truth conditioning tokens with model-generated ones during training at increasing rates, is one. But none of these approaches fully close the training-inference gap, and most large language models are still trained with teacher forcing for practical efficiency reasons.

A practical observation that follows from exposure bias is that hallucination rates are not uniform across a response. Hallucinations tend to cluster later in long responses rather than appearing uniformly throughout. This matches the theoretical prediction: early in a response, the model is conditioned largely on the correct prompt (which falls within the training distribution), so its outputs are more reliable. As the model generates more of its own tokens, the conditioning context shifts increasingly toward generated content, moving further from the training distribution and accumulating more divergence. For practical deployments, this suggests that very long generated responses should be treated with more skepticism on their later sections than on their early ones, all else being equal.

Exposure bias also interacts with the specific model architecture in non-obvious ways. Autoregressive models that use full causal attention are more susceptible to long-range error accumulation than models with shorter effective context windows, because a model with a large effective attention range can be influenced by early errors across many subsequent steps. This is one reason why some researchers have explored recurrent or state-space architectures as alternatives: models with more bounded memory may sometimes be more resilient to error propagation, even if they sacrifice performance on tasks requiring long-range coherence.