Hallucination Causes › Worked Example: Exposure Bias Accumulation

To make exposure bias concrete, consider a simplified example of how error accumulates across generation steps when a model is asked to produce a factual historical account.

Suppose the model is generating a paragraph about a specific historical event. At step 10, it generates an incorrect date: "1914" instead of the correct "1917." This incorrect date becomes part of the conditioning context for all subsequent tokens. The model now operates as if the event happened in 1914.

The correct year 1917 has high probability when conditioned on the correct prefix (the model has strong associations between the event's name and the year 1917 from training). But conditioned on a prefix that already states 1914, the model's distribution shifts: it may now amplify details that are more consistent with 1914 (other events from that year, the political climate of that period, figures who were prominent then) and suppress details associated with 1917. The hallucination cascades, not because the model lacks knowledge about 1917, but because the erroneous context now points in a different direction.

The formal probability divergence illustrates this. Let $y^*$ be the correct continuation and $\hat{y}$ be the continuation after an error at step $t$. We measure the divergence between the model's behavior in the two cases:

$$\Delta_t = D_{\text{KL}}\!\left(P(y_{t+1} \mid y_{1:t}^*) \;\|\; P(y_{t+1} \mid \hat{y}_{1:t})\right)$$

where:

• $\Delta_t$: the divergence at step $t$ between the distribution conditioned on correct context and the distribution conditioned on error context
• $D_{\text{KL}}(P \| Q)$: the Kullback-Leibler divergence from distribution $Q$ to distribution $P$, measuring how much $P$ differs from $Q$
• $P(y_{t+1} \mid y_{1:t}^*)$: the model's next-token distribution conditioned on the correct prefix up to step $t$
• $P(y_{t+1} \mid \hat{y}_{1:t})$: the model's next-token distribution conditioned on the error-containing prefix up to step $t$
• $y_{1:t}^*$ and $\hat{y}_{1:t}$: shorthand for the full prefix sequences $y_1^*, \ldots, y_t^*$ and $\hat{y}_1, \ldots, \hat{y}_t$ respectively

This divergence grows as the generated prefix diverges further from the true prefix. Since the model is never trained on diverged prefixes during teacher-forced training, $\Delta_t$ is not bounded by the training objective and can grow arbitrarily with sequence length. This is the mathematical reason why hallucination compounds in long-form generation.

The key insight from this worked example is that exposure bias does not require the model to "not know" the correct answer. The model may have reliable knowledge of the correct year in the abstract. The hallucination arises because the error changes the input distribution in a way that activates different, incorrect associations. This is why hallucination is not always fixed by giving the model more facts: sometimes the problem is not missing knowledge but corrupted context.