Hallucination Types in Language Models: A Complete Guide › Inconsistency

Inconsistency is a hallucination type that does not require checking against external sources: the model contradicts itself within a single response or across a conversation.

Inconsistency deserves recognition as a distinct hallucination type because it is detectable from the output alone, without any external knowledge, and it arises from a different mechanism than the other types we have discussed. While factual errors and fabrication stem from incorrect parametric knowledge or generative patterns, inconsistency reflects the model's lack of a persistent global state during generation. The model generates each token based on local context, without maintaining a structured representation of the claims it has already made. When different parts of a long response draw on the same underlying knowledge, small variations in how the context presents that knowledge can lead to contradictory outputs.

A model can contradict itself within a single long response. It might describe a character as a surgeon in one paragraph and "a specialist in internal medicine" in another, or state that a project was completed in 2022 early in a response and 2023 later. These inconsistencies arise partly because language models generate text left to right, without a global state tracking what they have already said. Each token is generated based on local context, which may not perfectly propagate constraints established earlier in the document.

Intra-document inconsistency is more likely in:

• Long-form generation (essays, reports, stories)
• Complex multi-step reasoning where intermediate conclusions constrain later ones
• Responses that cover multiple sub-topics with overlapping facts

The length of modern context windows has helped somewhat: longer context allows the model to "see" earlier statements when generating later ones. But even with large context windows, the model's attention is not uniformly distributed, and constraints set early in a long document can be effectively forgotten by the time later sections are generated. Attention patterns in transformers tend to focus on the most recent tokens and on certain salient anchor tokens earlier in the sequence, but they do not guarantee that every constraint established at position $i$ is propagated to every token generated at position $j > i$.

The problem is especially acute in structured outputs like reports or analyses, where early sections set up factual claims that later sections build on. If an early section introduces a number, name, or claim, and the model later contradicts it while building an argument, the result is internally inconsistent reasoning that may lead to wrong conclusions even if individual steps look plausible.

One useful way to think about intra-document inconsistency is as a manifestation of the model's lack of a working memory in the traditional sense. Human writers maintain a mental model of what they have said and check new claims against it. Language models have no such mechanism. The best approximation is that the context window itself is working memory, and when earlier content falls outside the attention's effective focus, it is as if the model has forgotten it.

In conversational settings, models can contradict information provided by the user or stated by themselves in an earlier turn. A user who says "I live in Berlin" in turn 1 might receive a response in turn 5 that assumes they live in a different city. Or a model that claims "I cannot access real-time information" in one turn might confidently describe a recent news event in the next.

Cross-turn inconsistency is a particular challenge for long conversations, where earlier context may be compressed or lost. It is also related to the well-documented tendency of models toward sycophancy: a model might initially state one position and then reverse it when the user pushes back, even if the original position was correct. In such cases the inconsistency is not random but systematically biased toward agreeing with the user.

The sycophancy-inconsistency connection is worth examining more carefully. A model trained with human feedback may learn that responses that agree with the user receive better ratings. Over many training examples, this can produce a systematic tendency to reverse factually correct claims when challenged, not because the model has found new evidence but because agreement generates higher reward. The resulting cross-turn inconsistency is not a failure of factual knowledge but a failure of calibration: the model's stated beliefs shift with social pressure rather than evidence. This has been documented in evaluations where evaluators push back on correct model responses and observe that the model frequently capitulates, restating the user's incorrect position as if it had always believed it.

Cross-turn inconsistency can also arise from context window limitations in long conversations. When a conversation exceeds the context window, earlier turns must be summarized or discarded. Information stated in discarded turns is unavailable to the model when generating later responses, leading to inconsistencies that are not the result of sycophancy but of context loss.

A subtler form of inconsistency occurs when the model does not follow through on what it claims it will do. A model might say "I will organize this into three sections: Background, Current State, and Future Outlook" and then proceed to generate content with completely different sections. Or it might promise to cite its sources and then provide none.

This is not always detectable without reading the response carefully, but it undermines user trust and task completion. It is related to the broader challenge of instruction following: models do not have a commitment mechanism that enforces consistency between a stated plan and subsequent output. If a model says "I will answer in exactly three bullet points" and then produces five, or says "I will avoid making claims I am uncertain about" and then proceeds with confident assertions, it has produced instruction-response inconsistency even if every individual claim is factually accurate.

Instruction-response inconsistency is a particular problem in multi-step or structured tasks where the model announces a plan at the start. The plan represents the model's prediction of what a good response looks like from the beginning of the response, before any content has been generated. But the actual generation process is driven by different forces, including the emerging content itself, and the announced plan may not survive contact with the actual generation dynamics. This gap between stated intentions and generated content is a frustration for users who are trying to direct model behavior through explicit plans and commitments.