Hallucination Mitigation: RAG, Decoding, and Training › Decoding Strategies

Hallucination is a generation-time phenomenon: a model that has encoded some degree of uncertainty about a fact may still produce a confident, incorrect token if the decoding procedure pushes it toward high-probability completions. Decoding-time interventions aim to modify this behavior without changing the model's weights. They are appealing because they can be applied to any model at any time, with no training cost, but their effect is limited by what the model has already learned.

The simplest decoding intervention is temperature. Standard next-token generation samples from:

$$P(x_t \mid x_{1:t-1}; T) = \frac{\exp(l_{x_t} / T)}{\sum_{x'} \exp(l_{x'} / T)}$$

where:

• $l_{x_t}$: the logit for token $x_t$, the raw unnormalized score from the model's output layer
• $T$: the temperature parameter controlling distribution sharpness
• $x_{1:t-1}$: the sequence of tokens generated so far (the context)
• The denominator sums exponentials over all vocabulary tokens $x'$

When $T < 1$, the distribution sharpens: the highest-probability tokens become even more probable relative to alternatives. When $T > 1$, the distribution flattens, increasing diversity but also increasing the probability of low-probability tokens including incorrect ones.

For factual generation tasks, low temperature is generally preferable. The model's most probable token at each step is more likely to be the one best supported by training evidence. However, the relationship is not monotone: greedy decoding ($T \to 0$, always selecting the argmax) can amplify repetition and get stuck in degenerate loops, and it does nothing to prevent the model from generating a confident wrong token if that token has the highest logit.

A useful mental model for temperature: think of the model's logit distribution at each step as representing its "belief" about the next token. Low temperature sharpens the model's expressed beliefs, making it commit more firmly to its top choice. High temperature softens those beliefs, making it hedge toward alternatives. For factual accuracy, you want the model to commit firmly to its best-supported beliefs, which argues for lower temperature. But if the model's best-supported belief is wrong, lower temperature makes that error more pronounced.

Top-p sampling (nucleus sampling) restricts generation to the smallest set of tokens whose cumulative probability exceeds a threshold $p$. Formally, the nucleus $V_p$ is defined as:

$$V_p = \min \left\{ V' \subseteq V : \sum_{x \in V'} P(x) \geq p \right\}$$

where:

• $V$: the full vocabulary of possible tokens
• $V'$: a candidate subset of the vocabulary, sorted by descending probability
• $p$: the cumulative probability threshold (e.g., 0.9 means keep the smallest set of tokens that together account for 90% of the probability mass)
• $V_p$: the chosen nucleus, the smallest such subset meeting the threshold

We sample uniformly over $V_p$ after renormalizing its probabilities to sum to 1. Top-k sampling instead restricts to the $k$ highest-probability tokens regardless of their cumulative probability.

These filters prevent the model from sampling extremely low-probability tokens that might include hallucinated content. They do not prevent high-confidence hallucinations, but they do reduce the tail risk of the model generating tokens that are essentially noise in the distribution. The intuition is that truly hallucinated content often requires the model to produce tokens that are improbable given the context: setting $p = 0.9$ ensures you never sample from that improbable tail.

The difference between top-p and top-k is important in practice. Top-k truncates to a fixed number of tokens regardless of how concentrated the distribution is. If the model is very confident (one token has 95% probability), top-k with $k=50$ still admits 49 low-probability alternatives. Top-p dynamically adjusts: when the distribution is concentrated, the nucleus is small (perhaps just 2-3 tokens), and when it is diffuse, the nucleus is large. For hallucination mitigation, top-p is generally preferable because it is responsive to the model's actual confidence at each step.

Self-consistency is one of the most empirically reliable decoding strategies for improving factual accuracy. The procedure is straightforward:

1. Generate $N$ independent responses to the same prompt using sampling (not greedy decoding)
2. Aggregate the responses to find the most consistent answer

For questions with extractable answers (factoid questions, numerical answers, reasoning chains), you identify the answer that appears most frequently across the $N$ responses. The key insight is that the correct answer, when the model knows it, tends to be consistently generated across samples, while hallucinated content tends to vary.

Why does this work? When the model has strong training signal for a fact, the underlying probability distribution assigns high probability to the correct completion, and different samples drawn from that distribution tend to land on the same answer. When the model is uncertain or guessing, the distribution is flatter, and samples diverge. Self-consistency exploits this asymmetry: agreement across samples is evidence of stored knowledge, while disagreement is evidence of uncertainty.

The aggregation step works differently depending on task type:

• Closed-form answers: majority vote across the $N$ responses
• Numerical answers: median or mode of the extracted values
• Reasoning chains: majority vote on the final conclusion, using the most common chain as the output
• Open-ended generation: selecting the response with the highest average token probability, which is a proxy for the model's confidence

Self-consistency was originally proposed as a prompting technique for chain-of-thought reasoning but generalizes broadly. Its hallucination-reduction effect comes from averaging out the variance in the model's generation. A single sample might land on an incorrect token; the average of many samples is pulled toward the model's learned central tendency.

The cost of self-consistency is compute: generating $N$ responses is $N$ times more expensive than generating one. For latency-sensitive applications, this is often prohibitive. For accuracy-critical applications where the query can be processed asynchronously, it is a practical approach. There are also efficient variants that stop sampling early when consensus is reached after $k < N$ samples, reducing average cost on high-confidence queries.

More targeted decoding interventions modify the token probability distribution itself based on external factuality signals. These go beyond simple temperature and sampling adjustments to actively reshape what the model generates.

Contrastive decoding computes the difference between a capable large model and a smaller model for each candidate token. The reasoning is that tokens where the large model assigns much higher probability than the small model tend to be the tokens the large model has specifically learned to prefer, as opposed to superficial continuations that both models would predict. Formally, for a capable model $M_{\text{large}}$ and an amateur model $M_{\text{small}}$, the contrastive score for token $x$ is:

$$s_{\text{CD}}(x) = \log P_{M_{\text{large}}}(x) - \log P_{M_{\text{small}}}(x)$$

where:

• $s_{\text{CD}}(x)$: the contrastive decoding score for token $x$
• $P_{M_{\text{large}}}(x)$: the probability assigned to $x$ by the capable (large) model
• $P_{M_{\text{small}}}(x)$: the probability assigned to $x$ by the amateur (small) model

Tokens that both models consider likely (common continuations, function words) get low contrastive scores. Tokens the large model uniquely prefers get high scores. This acts as an amplification of what the large model has specifically learned. For factuality, the intuition is that small models have weaker factual knowledge, so facts the large model knows strongly will have high contrastive scores. The method has shown meaningful improvements on knowledge-intensive tasks and is relatively low-cost since the small model is cheap to run.

A practical limitation of contrastive decoding is that it requires running two models in parallel, doubling inference complexity. For latency-sensitive applications, this may be unacceptable. There are also cases where the small model assigns high probability to the correct token for the wrong reason (e.g., because the token is a common word), which reduces the signal quality of the contrastive score.

Inference-time factuality steering uses linear probes trained to detect factual versus confabulated content in the model's residual stream. At inference time, the probe's gradient direction can be added to the model's internal representations to steer generation toward factually supported completions. This is an active research area and not yet widely deployed, but represents a direction toward intrinsic factuality control that operates below the level of token probabilities.

Retrieval-constrained decoding combines RAG with decoding-time constraints. Rather than simply placing retrieved documents in context, you restrict the generation vocabulary at each step to tokens that are consistent with at least one retrieved document. This enforces hard faithfulness but requires online retrieval during each decoding step, which is computationally expensive. In practice, this approach is most useful for highly structured generation tasks (e.g., generating database queries or code) rather than open-ended text generation.

Chain-of-thought prompting, which asks the model to reason step-by-step before giving a final answer, has a beneficial side effect on factuality beyond its primary benefit for multi-step reasoning. When a model must articulate its reasoning process, errors become more visible and correctable.

The factuality benefit of chain-of-thought comes from several mechanisms. First, the reasoning chain exposes intermediate claims that can be checked against retrieved documents or common sense. A response that states "Napoleon was born in 1769 in Corsica, and since Corsica became French territory in 1768, he was a French citizen" is more verifiable than a bare assertion "Napoleon was French." Second, the act of articulating reasoning chains may itself constrain generation: a model that has just stated a fact in the reasoning chain is more likely to use it correctly in subsequent steps. Third, longer reasoning chains give retrieval-augmented systems more opportunities to match and retrieve relevant documents before the final answer is generated.

The failure mode of chain-of-thought for factuality is that confident but incorrect reasoning chains can lead the model to a wrong answer with apparent logical support. A fluent incorrect reasoning chain is harder to reject than a bare incorrect assertion, because it pattern-matches to the surface form of sound reasoning. Users may be more convinced by a confidently wrong argument than by a confidently wrong bare assertion.