Hallucination Types in Language Models: A Complete Guide › Quantifying Hallucination Types

Each hallucination type requires a different evaluation approach. Researchers have developed a range of metrics and benchmarks to measure hallucination at scale.

For intrinsic hallucinations, natural language inference is the dominant automated approach. NLI models are trained to classify whether one sentence (the hypothesis) is entailed by, neutral with respect to, or contradicted by another sentence (the premise). This maps naturally onto hallucination detection:

• Premise: a sentence from the source document
• Hypothesis: a sentence from the generated summary
• Entailed: the summary sentence is supported by the source (faithful)
• Contradicted: the summary sentence contradicts the source (intrinsic hallucination)
• Neutral: the summary sentence neither follows from nor contradicts the source (possibly extrinsic)

Tools like SummaC, FactCC, and MiniCheck use this approach to score summary faithfulness. They decompose the output into atomic claims and check each claim against the source, producing a per-claim faithfulness score. SummaC in particular introduced the insight that the faithfulness score should be computed at the sentence level rather than the document level, because document-level NLI tends to be dominated by the longest or most salient claim and misses sentence-level contradictions.

The NLI-based approach has an important limitation: it can only detect contradictions between claims that share significant lexical overlap with source sentences. Abstract or paraphrased claims that are semantically contradicted but lexically distant from the source may receive neutral or even entailment scores from weaker NLI models. More capable cross-encoder models like DeBERTa-based NLI systems handle paraphrase better, but the gap is not eliminated. This is one reason why research has moved toward stronger model-based evaluators for faithfulness assessment.

For open-domain factual errors, researchers use benchmarks that test factual accuracy against ground truth.

TruthfulQA contains 817 questions specifically designed to elicit common misconceptions and hallucinations, covering health, law, finance, and politics. The benchmark is notable for its adversarial design: the questions target claims that are commonly believed but false, or that exploit known model weaknesses. The benchmark revealed that larger models sometimes perform worse than smaller ones on specific misconception types, because larger models are more fluent and more likely to produce convincing-sounding wrong answers.

HaluEval is a large-scale evaluation benchmark with human-annotated hallucinations across QA, dialogue, and summarization, containing over 35,000 examples. Its breadth across task types makes it useful for comparing hallucination rates across settings and for training hallucination detection models.

FActScoring takes a different approach: it decomposes biography generation into atomic facts and verifies each against a reference corpus (typically Wikipedia). This decomposition strategy is important because it allows fine-grained measurement rather than a single per-document score. A model might generate a ten-sentence biography where eight sentences are fully accurate and two contain factual errors. A document-level score conflates these, while FActScoring isolates the two problematic claims.

FEVER (Fact Extraction and VERification) provides 185,445 claims paired with Wikipedia evidence, allowing automated fact verification at scale. It is used both as a benchmark and as training data for fact-verification models.

As LLMs have become more capable, another approach has emerged: using a strong language model as an evaluator for the outputs of another. The evaluating model is given the source (or a reference fact base) and the output, and asked to identify factual inconsistencies. This approach is flexible and does not require task-specific training, but it introduces the question of whether the evaluating model is itself reliable. We will return to this tension in the Hallucination Detection chapter.

Model-based evaluation has become increasingly common under the rubric of "LLM-as-judge." The key practical advantage is coverage: a strong LLM can evaluate hallucinations across all types, including extrinsic and fabrication types that NLI-based methods cannot reliably detect. The key limitation is that the evaluator model may have the same knowledge gaps or the same tendency to hallucinate as the model being evaluated, and there is no guarantee that its judgments are accurate when both the evaluated model and the evaluator have incorrect beliefs about the same topic.

The following heatmap shows how well each detection method covers each hallucination type, giving you a quick reference for choosing evaluation strategies.