Hallucination Types in Language Models: A Complete Guide › Hallucination Benchmarks

Researchers have developed standardized benchmarks to evaluate hallucination across settings:

TruthfulQA is particularly notable because its questions are specifically designed to target common misconceptions, the kinds of things language models are likely to have absorbed from their training data. Questions probe the model's ability to resist generating plausible-but-false completions in domains where misconceptions are common. The benchmark revealed that larger models are sometimes more likely to produce convincing hallucinations, not less, because they are better at generating fluent text that sounds authoritative. This counterintuitive result reflects a real tension: a more capable model generates more convincing text, and a more convincing hallucination is more dangerous than an obviously wrong one.

HaluEval covers multiple task types and includes both automatic and human evaluation. It shows that hallucination rates vary significantly by task: summarization and open-ended QA show higher hallucination rates than more constrained extraction tasks. The variation across tasks is an important practical finding: deploying a model on tasks where hallucination rates are known to be high should trigger additional safeguards.

FEVER is notable for its scale and systematic annotation methodology. Because claims are paired with Wikipedia evidence, automated fact verification can be applied at scale, making it useful both as a benchmark and as training data for verification models. The dataset has proven foundational for research on fact-checking systems that operate at production scale.