Hallucination Causes › The Interaction of Causes

In practice, these four causes do not operate in isolation. They interact and reinforce each other in ways that make the hallucination problem harder than any single cause would suggest.

Consider what happens when all four operate simultaneously: a model trained on noisy web data (training data issues) encounters a question about a tail entity (knowledge gap). It generates a confident-sounding response (completion pressure) in a long-form answer where an early error compounds (exposure bias). Each cause amplifies the others. The noisy training data means the model has unreliable signal for the tail entity. The knowledge gap means the model must extrapolate. The completion pressure means it will not abstain. The exposure bias means that whatever error it commits early in the response will cascade through the rest.

This interaction also explains why hallucination patterns vary systematically across model sizes and training configurations. Larger models trained on more data tend to hallucinate less on high-frequency facts: their stronger signal for well-attested entities reduces the training data issue and the knowledge gap problem. But they may not reduce hallucination on tail entities proportionally, because scale helps only where data is already sufficient. And larger models can be better at fluent, confident generation, which may paradoxically worsen completion pressure and overconfident priors.

The interaction between causes also explains why mitigation strategies that address only one cause produce partial results. Retrieval augmentation (adding external knowledge at inference time) addresses knowledge gaps but leaves exposure bias and completion pressure untouched. Careful data curation addresses training data issues but cannot retroactively fill knowledge gaps for entities that never appeared in sufficient quantity. Better uncertainty calibration through RLHF addresses the surface expression of completion pressure but does not change the underlying lack of a world model or the exposure bias structure. These interventions complement each other rather than substitute for each other.

Understanding the interaction of causes also changes how you should think about evaluation. A benchmark that tests only high-frequency factual questions will not reveal tail entity hallucination. A benchmark that tests only short responses will not reveal exposure bias accumulation. A benchmark that only measures whether answers are correct or incorrect will not reveal miscalibration in uncertainty expression. Complete hallucination evaluation requires explicitly probing each of the four causes, because a model can perform well on one dimension while failing badly on another.

The interaction of causes is also sensitive to model scale in ways that are not always intuitive. Scaling up model size and training data simultaneously tends to reduce hallucination on well-documented facts because larger models have greater capacity to memorize and accurately recall high-frequency information. But scaling also increases the fluency and coherence of generated text, which means that hallucinations, when they do occur, tend to be more convincing and harder to detect. A small model's hallucination may be obviously wrong or awkwardly phrased, making it easy to catch. A large model's hallucination may be perfectly fluent, internally consistent within the generated text, and superficially plausible, making it much more dangerous for users who cannot independently verify the claims. The very property that makes large models useful (fluent, coherent, confident generation) is also what makes their hallucinations more harmful.

This creates a critical calibration challenge as models improve. Progress in language modeling capability, as measured by perplexity or instruction-following quality, does not automatically translate into progress in hallucination reduction. A model that is much better at following instructions and producing fluent prose may hallucinate at similar rates as a simpler model on tail entities, while producing hallucinations that are far more convincing. Evaluating models on hallucination separately from general capability is therefore essential, and using hallucination metrics that account for the deceptiveness of the errors (not just their frequency) is important for capturing the practical risk.