Hallucination Mitigation: RAG, Decoding, and Training › Retrieval-Augmented Generation

The most widely deployed hallucination mitigation technique is retrieval-augmented generation, or RAG. The core intuition is simple: if a model lacks reliable factual knowledge from training, you can supply that knowledge at inference time by retrieving relevant documents and placing them in the context window. The model then generates its answer based on what you retrieved, rather than relying on potentially inaccurate parametric memory.

To understand why this works so well, consider what hallucination is: the model generating tokens that are statistically plausible given its training but that do not correspond to facts about the world. The model has no dedicated "fact retrieval" module; facts are distributed across billions of floating-point parameters, blended with patterns, styles, and associations learned from everything else in the training corpus. Asking the model to recall a specific fact is asking it to reconstruct a signal from noisy, distributed weights. Retrieval sidesteps this problem entirely: instead of reconstructing a fact from memory, the model reads it.

We have covered RAG architecture in depth in the retrieval-augmented generation chapters of Part XLIV: Retrieval-Augmented Generation. The focus here is specifically on how RAG reduces hallucination, its failure modes as a mitigation strategy, and implementation details relevant to factuality.

A language model's parametric knowledge is frozen at training time, probabilistic in nature, and unverifiable during inference. Retrieved documents have none of these limitations: they are up-to-date, explicit, and attributable to a source.

When you retrieve a document containing the answer and place it in the context window, the model faces a fundamentally different generation problem. Rather than reconstructing a fact from statistical weights learned over billions of tokens, it reads the answer from text that is immediately present. For most factual question types, surface reading comprehension is far more reliable than parametric recall. A model that hallucinates the founding year of a company when asked cold will almost certainly answer correctly if you provide a Wikipedia excerpt that states the founding year in the first paragraph.

The mechanism generalizes beyond simple lookups. RAG improves factuality by:

• Grounding answers in verifiable sources: The model cannot assert something the retrieved document contradicts without the user noticing. This external check constrains confabulation even when the model would otherwise have interpolated from nearby facts.
• Reducing knowledge gap exposure: Many hallucinations occur precisely when the model is asked about tail entities or recent events outside its training data. Retrieval plugs these gaps directly.
• Enabling attribution: When the answer comes from a retrieved document, the model can cite the source. Attribution lets users verify claims and increases accountability.
• Making freshness tractable: A model's training data has a cutoff, but a retrieval index can be updated continuously. For questions about recent events or rapidly changing information, retrieval is the only viable path to accurate answers without retraining the model.

The quality of RAG as a hallucination mitigation strategy depends heavily on three design choices: the retrieval model (how well it finds the right documents), the knowledge base (whether the right documents exist and are well-curated), and the context formatting (how retrieved passages are presented to the generation model). Improving any of these three levers improves factuality, but they involve different engineering investments and have different failure signatures.

RAG does not eliminate hallucination. Understanding its failure modes is essential for deploying it responsibly.

Retrieval failure is the most fundamental: if the relevant document is not retrieved, the model falls back on parametric memory. This happens when the query-document similarity is not well-captured by the embedding model, when the knowledge base does not contain the relevant document at all, or when the question requires synthesizing across many documents that together exceed context limits. Retrieval failure is silent: the model silently falls back to parametric generation, often without indicating that the context did not contain the relevant fact.

Context ignoring occurs when the model retrieves the correct document but generates an answer inconsistent with it. This failure mode is more common than expected. Models trained on enormous corpora develop strong prior beliefs about what the answer should be. When the retrieved document contradicts the model's parametric knowledge, the model sometimes defaults to its prior rather than faithfully reading the document. This is especially pronounced for commonly misrepresented facts, where training data provides overwhelming but incorrect signal. A model that "knows" a popular misconception may override a retrieved correction.

Faithful summarization with unfaithful extension is subtle. The model correctly reports what the retrieved document says, then continues generating text that goes beyond the document, interpolating from parametric memory without marking the boundary between retrieved and generated content. The answer appears fully grounded but contains unverified claims. This is particularly dangerous because the grounded portion gives the response a surface appearance of reliability.

Noise sensitivity affects retrieval when documents are long or topically broad. A retrieved passage that contains the relevant fact also contains irrelevant text. Models are sensitive to position (facts early in context are more reliably used than facts buried deep) and to distraction (irrelevant but plausible text can nudge generation away from the correct answer). The "lost in the middle" phenomenon describes how models attend primarily to the beginning and end of their context window, often missing facts placed in the middle of long retrieved passages.

Multi-hop reasoning failures emerge when answering a question requires combining facts from multiple retrieved documents. The model must retrieve both pieces of information, identify that they need to be combined, and reason across them correctly. Each step introduces failure probability, and the compounding of these probabilities means that multi-hop RAG is substantially less reliable than single-document RAG. For complex questions, this limitation is significant.

Several techniques improve the probability that a model will faithfully use retrieved context rather than overriding it with parametric memory. The underlying challenge is that instruction-following is imperfect: a model told to "use only the provided documents" has not been surgically modified to ignore its training; it has been given a natural language instruction that it may or may not prioritize over its prior beliefs. The techniques below increase the probability of faithful behavior, but none guarantee it.

The most effective prompt-level technique is an explicit instruction to answer only from the provided context:

This framing shifts the task from "use documents as a hint" to "read from documents exclusively." The improvement is real but incomplete: models still occasionally override context, especially when parametric priors are strong.

Contrastive prompting goes further by explicitly informing the model that the context should override its priors:

Self-consistency with context checking addresses the context-ignoring failure mode computationally. Generate multiple responses, then check each response against the retrieved document using an entailment model or another LLM. Responses that contradict the retrieved document are filtered out, and the most consistently faithful response is selected. This adds latency but significantly reduces context-ignoring failures.

Citation forcing is a structural technique: require the model to cite the specific sentence in the retrieved documents that supports each claim it makes. A model that cannot cite a supporting sentence should not make the claim. This makes violations of faithfulness detectable: if the model makes a claim and its cited sentence does not support it, the system can flag or reject the response. Citation forcing also helps users understand which parts of the answer come from which documents, improving interpretability.

Chunk granularity tuning addresses the noise sensitivity failure mode. Rather than retrieving entire documents or large passages, retrieve fine-grained chunks (sentences or short paragraphs) that are more likely to contain the specific fact needed. Smaller chunks reduce distraction from irrelevant content but may miss context that spans sentences. The optimal chunk size depends on the nature of the queries and documents in your deployment.