Hallucination Detection: NLI, Self-Consistency & Learned Models › Learned Detection Models

Rather than reasoning from first principles about entailment or consistency, a different approach is to train a dedicated model specifically for the task of detecting hallucinations. These learned detectors treat factual inconsistency detection as a supervised classification problem, using human-annotated or synthetically generated training examples to learn what factual errors look like.

The general setup is:

• Input: A source document $D$ (or context) and a generated claim $c$
• Output: A score or binary label indicating whether $c$ is supported by $D$
• Training signal: Human-annotated examples of supported and unsupported claims, or synthetically generated perturbations of true claims

The appeal of this approach is that the learned detector can exploit patterns that are specific to how language models fail, which may not be fully captured by general NLI reasoning. A model trained on thousands of examples of LLM hallucinations will develop representations tailored to the distribution of real hallucination errors, rather than the distribution of NLI training data (which often consists of human-written sentences with artificial contradictions designed to probe logical reasoning). The distribution shift between human-authored NLI examples and LLM hallucinations is significant enough that dedicated training can yield meaningful accuracy gains.

FactCC (Factual Consistency Checking) was one of the first dedicated models for summarization factuality. It uses a BERT-based classifier trained on synthetic data generated by perturbing source sentences in specific ways designed to mimic the failure modes of summarization models:

• Entity swap: Replace a named entity with a different entity of the same type, simulating the common error of attributing actions or properties to the wrong person or organization
• Pronoun swap: Change pronouns to introduce coreference errors that flip the subject of an action
• Number perturbation: Modify numerical values slightly, simulating the tendency of summarization models to change quantities while keeping surrounding text intact
• Noise injection: Insert or delete words at random, creating grammatically plausible but factually inconsistent text

These perturbations create realistic-looking hallucinations that share the surface form and vocabulary of the original text. The model learns to distinguish original sentences (factually consistent) from perturbed ones (hallucinations). The key advantage is that synthetic data generation is cheap, allowing for large-scale training without expensive human annotation.

FactCC formulates detection as binary classification. Given a (source, claim) pair $(D, c)$, the model predicts a consistency score:

$$\hat{y} = \sigma\!\left(W \cdot \text{BERT}([D;\, \text{SEP};\, c])_{\text{CLS}}\right)$$

where:

• $[D;\, \text{SEP};\, c]$: the source document and claim concatenated with a separator token, fed as a single input sequence
• $\text{BERT}(\cdot)_{\text{CLS}}$: the CLS token representation output by BERT after processing the full concatenated input, which encodes a summary of the relationship between $D$ and $c$
• $W$: a learned weight matrix that maps the CLS embedding to a scalar logit
• $\sigma$: the sigmoid function, which maps the logit to a probability $\hat{y} \in (0, 1)$
• $\hat{y}$ near 1 means the model predicts the claim is factually consistent with the source

Training uses binary cross-entropy loss over the labeled examples. The model learns to associate the entity swaps, number perturbations, and other synthetic errors with the "inconsistent" label, building up internal representations that capture what factual disagreement looks like. The resulting model transfers reasonably well to real summarization hallucinations because real errors frequently take the same forms as the synthetic perturbations.

A more recent approach, MiniCheck, addresses the practical challenge that larger NLI models are expensive to run at scale. MiniCheck fine-tunes a small language model (7 billion parameters or fewer) specifically for the task of claim verification against a grounding document, achieving performance competitive with GPT-4 on factuality benchmarks at a fraction of the cost.

MiniCheck casts the problem as a yes/no question: "Is this claim supported by the document?" The model processes the source document and claim together and outputs a binary probability. It is trained on a curated mixture of factual consistency datasets from summarization, question-answering faithfulness datasets, and synthetic perturbations similar to those used in FactCC.

By combining diverse supervision signals, MiniCheck generalizes better across domains than models trained on a single dataset. It is also designed to handle long-form inputs using a windowed approach when source documents exceed the model's context window: the source is split into overlapping windows, each window is scored independently against the claim, and the maximum entailment score across all windows is reported. This windowed design mirrors the logic of SummaC but uses a single model for both the window comparison and the final scoring step.

The cost advantages of MiniCheck are substantial. Replacing GPT-4-based factuality checks with MiniCheck in a production pipeline reduces inference cost by roughly two orders of magnitude, making it practical to check every generated response rather than only sampling a subset.

TRUE (Towards a Unified Model for Factual Precision Evaluation) trains a single model on many fact-checking tasks simultaneously to build a general-purpose factuality detector that works across diverse input types. The training datasets include:

• Summarization factuality (XSum, CNN/DailyMail) covering news summaries
• Dialogue factuality (Wizard of Wikipedia, FaithDial) covering conversational grounding
• Question answering faithfulness (NaturalQuestions, TriviaQA) covering open-domain QA
• Fact verification (FEVER) covering explicit claim verification against Wikipedia

TRUE frames all tasks as textual entailment and uses a shared NLI objective. The motivation is that these tasks are all instances of the same underlying question, "Does text A support text B?", but they vary in their input structure, domain, and the nature of the claims being checked. Training on a mixture forces the model to learn a general notion of factual support rather than overfit to the stylistic patterns of any single dataset.

The multi-task training signal acts as a form of regularization. A model that can perform well on both news summarization and dialogue grounding must have learned more fundamental representations of factual consistency rather than surface features that correlate with one domain's error patterns. On the TRUE benchmark, a model trained on the full mixture outperformed models trained on individual tasks across all evaluation domains, showing that the multi-task training provides a measurable benefit rather than just averaging the performance of individual models.

TRUE is particularly valuable when you are building a system that must handle diverse input types, such as a general-purpose LLM assistant that answers questions, summarizes documents, and engages in dialogue. A domain-specific model like FactCC would be optimized for only one of these use cases.

A fourth paradigm that has emerged alongside learned detector models is using large language models themselves as judges of factual consistency. Rather than training a dedicated classifier, you prompt a powerful LLM to evaluate whether a claim is supported by a source document or is factually correct.

A typical prompt follows this structure:

This approach has several advantages. It requires no fine-tuning, can be applied immediately to new domains without any labeled data, and can produce natural language explanations of why a claim is flagged. Larger LLMs like GPT-4 have shown strong correlation with human judgments on factuality benchmarks, often outperforming smaller dedicated models.

The disadvantages are cost and reproducibility. Each evaluation requires a full LLM forward pass, which is expensive at scale. Results also depend on the specific prompt and model version, making them hard to reproduce exactly. There is also a concern about using a model to evaluate the outputs of the same model family, which can introduce systematic blind spots if both models share the same factual errors.

Despite these limitations, LLM-as-judge has become the de facto evaluation approach for factuality in many research papers and production systems, particularly when the goal is measuring aggregate quality across a large set of responses rather than real-time per-response detection.