Hallucination Mitigation: RAG, Decoding, and Training › Training Approaches

Decoding interventions and retrieval augmentation work at inference time without touching the model's weights. Training approaches go deeper: they modify what the model has learned to do, making it intrinsically less likely to hallucinate. Training-based mitigations are more powerful and persistent than inference-time interventions, but they are also more expensive and require data.

The most direct training approach is supervised fine-tuning (SFT) on high-quality, fact-checked datasets. If the model has learned to generate confident falsehoods because it was trained on noisy web data, training it additionally on verified content can help recalibrate its factual generation.

This works better for domain-specific deployment than for general factuality. A model fine-tuned on carefully curated medical literature will generate more accurate medical text because the training signal consistently reinforces the correct facts. A model fine-tuned on news articles will be more reliable on current events covered in that dataset. The improvement is real and often significant in the target domain.

The limitation is coverage: SFT on factual data can only improve the model's accuracy on facts present in the fine-tuning set. It does not improve the model's ability to recognize when it is outside its knowledge, and it can reduce diversity in a way that makes the model overconfident about the fine-tuning domain while no better on everything else. There is also the risk of catastrophic forgetting, as we discussed in the Finetuning Fundamentals chapters: heavy fine-tuning on a narrow factual domain can degrade the model's performance on unrelated tasks.

A practical application of SFT for factuality is training on examples that demonstrate appropriate uncertainty expression. Rather than training only on examples of correct answers, you include training examples where the correct output is an explicit "I don't have reliable information about this" rather than an attempted answer. This teaches the model that refusal is a valid and valued response, counteracting the tendency of instruction-tuned models to always attempt an answer.

RLHF trains a reward model on human preference judgments and then uses reinforcement learning to optimize the language model to produce outputs that would receive high reward. When human preferences include factual accuracy as a criterion, RLHF can reduce hallucination by directly rewarding non-confabulation.

The process works in three stages. First, human annotators rate pairs of model outputs on multiple criteria including helpfulness, safety, and factual accuracy. These ratings become training data for a reward model, which learns to predict a scalar preference score for any given (prompt, response) pair. Second, the language model is fine-tuned using RL (typically Proximal Policy Optimization, or PPO) to maximize the expected reward score, with a KL divergence penalty that prevents the model from drifting too far from its original behavior:

$$\mathcal{J}_{\text{RLHF}}(\theta) = \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot \mid x)} \left[ r(x, y) - \beta \, \text{KL}\!\left(\pi_\theta(\cdot \mid x) \;\|\; \pi_{\text{ref}}(\cdot \mid x)\right) \right]$$

where:

• $\pi_\theta$: the language model policy being optimized, with parameters $\theta$
• $\pi_{\text{ref}}$: the reference (base) policy before RLHF training
• $r(x, y)$: the reward model's score for response $y$ to prompt $x$
• $\beta$: the KL penalty coefficient, controlling how far the model can deviate from its reference
• $\mathcal{D}$: the distribution of prompts

When human annotators are instructed to penalize hallucinated or unverified claims, the reward model implicitly encodes a preference for factuality, and the RL training pushes the language model toward outputs that avoid false claims.

InstructGPT and subsequent models (GPT-4, Claude, Gemini) use RLHF with explicit instructions to annotators to prefer factually accurate responses and to downgrade responses containing false claims. The resulting models are measurably less likely to assert confidently false statements compared to base models fine-tuned on instruction data alone.

The mechanism is indirect: the reward model learns to predict what humans prefer, and humans prefer accurate responses, so the language model is pushed toward accuracy. But the reward model is itself imperfect, and annotators make errors or disagree, so the improvement is real but not absolute. RLHF does not eliminate hallucination; it reduces it probabilistically.

A deeper limitation is that RLHF provides no explicit signal about which facts are true and which are false. It only rewards what humans rate highly, and human raters cannot always verify factual claims in the responses they are rating. This means RLHF works better at eliminating obvious, verifiable falsehoods than at correcting subtle factual errors in specialized domains where annotators lack expertise. A medical hallucination that reads confidently and coherently may receive high human ratings even if it is incorrect.

DPO is a more stable alternative to RL-based RLHF. Rather than training a separate reward model and using RL, DPO directly optimizes the language model to prefer responses chosen by humans over responses rejected by humans, using a classification-style loss:

$$\mathcal{L}_{\text{DPO}} = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \left[ \log \sigma \left( \beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} \right) \right]$$

where:

• $(x, y_w, y_l)$: a training triple of prompt $x$, preferred response $y_w$ (won), and rejected response $y_l$ (lost)
• $\pi_\theta$: the language model being trained (with parameters $\theta$)
• $\pi_{\text{ref}}$: the reference (base) policy before DPO fine-tuning
• $\beta$: a temperature parameter controlling how far $\pi_\theta$ can deviate from $\pi_{\text{ref}}$
• $\sigma$: the sigmoid function

The loss maximizes the probability that the model assigns higher likelihood to the preferred response than the rejected response, relative to the reference model. This is equivalent to doing implicit reward learning and policy optimization in a single step, without ever materializing a separate reward model.

DPO has been applied to factuality by constructing preference datasets where the preferred response is a factually correct answer and the rejected response is a hallucinated one. This directly teaches the model to prefer truthful outputs over confabulated ones. The key requirement is a high-quality preference dataset with pairs of (truthful answer, hallucinated answer) for the same prompt. Constructing these pairs at scale requires either human annotation or automated hallucination generation (deliberately eliciting hallucinations from a base model to create negative examples).

A practical advantage of DPO over RLHF for factuality fine-tuning is stability: the optimization landscape is smoother, training is less sensitive to hyperparameters, and there is no risk of reward hacking (where the policy finds ways to maximize the reward model's score without being more factual). For domain-specific factuality improvements, DPO with a curated preference dataset is often the most effective and tractable training approach.

A related approach pairs fine-tuning with explicit knowledge supervision. In knowledge-grounded training, each training example includes both the question and the relevant supporting documents, and the model is trained to generate answers conditioned on those documents. The goal is to teach the model to rely on provided context rather than parametric memory.

This is effectively training a model to do RAG well, rather than retrofitting RAG onto a model trained without it. Models trained with knowledge-grounded objectives learn to distinguish "I know this from training" from "I am reading this from context," which is a skill that improves their behavior in RAG deployments. A model trained on (question, document, answer) triples learns to use the document as its primary information source. When deployed with retrieval augmentation, this training makes the model more faithful to retrieved context and less likely to override it with parametric priors.

ROME and MEMIT are surgical fact-editing approaches at the opposite extreme from full fine-tuning. Rather than retraining broadly, they locate specific factual associations stored in transformer feed-forward layers and directly modify those weights. The feed-forward layers in transformer blocks act as key-value memories: they store factual associations of the form "subject entity $\to$ attribute value," and these associations can be located and modified with relatively small weight changes.

Editing "The Eiffel Tower is in Paris" to "The Eiffel Tower is in Berlin" (as a test of the method) requires changing only a small number of weights in mid-layer feed-forward networks. ROME edits one fact at a time; MEMIT extends this to batch editing of thousands of facts simultaneously. The editing procedure involves:

1. Identifying which attention heads and feed-forward neurons are responsible for recalling the target fact
2. Computing the update direction in weight space that shifts the association from the old value to the new one
3. Applying the weight update with minimal impact on other stored facts

Fact editing is appealing for knowledge-update use cases (updating a model's beliefs about recent events without full retraining) but carries significant risks. Edited knowledge can interfere with nearby associative facts in ways that are hard to predict. Editing can create inconsistencies the model cannot reason about: the model may correctly report the edited fact in direct queries while continuing to use the old fact in indirect reasoning. There is limited understanding of what other beliefs change implicitly when a targeted fact is modified. For production hallucination mitigation, fact editing is currently more useful as a research tool than a deployment strategy.