Hallucination Types in Language Models: A Complete Guide › Self-Consistency as an Inconsistency Detector

Beyond NLI-based faithfulness checking, self-consistency offers a complementary approach that does not require a source document and is particularly suited to detecting inconsistency-type hallucinations. The basic idea is simple: if a model's answer to a question is consistent regardless of how the question is phrased or what sampling temperature is used, it probably knows the answer. If the answers vary, the model is uncertain, and inconsistent answers signal a likely hallucination.

Formally, the self-consistency approach generates multiple responses to the same prompt and measures agreement across responses. For factual questions with discrete answers, this means checking whether the most common answer agrees with the full distribution of answers. For more open-ended generation, it means checking whether key factual claims in one response are also present in other responses.

The self-consistency signal is interpretable: a claim that appears in 9 out of 10 independently generated responses is likely grounded in the model's parametric knowledge. A claim that appears in only 2 out of 10 responses is uncertain, and a response that contains it should be treated with skepticism. This does not guarantee correctness (the model could consistently hallucinate the same wrong answer), but it provides a probabilistic reliability signal that has proven useful in practice.