Reward Hacking › Mitigation Strategies

Given the inevitability of reward hacking under naive optimization, the RLHF pipeline incorporates several mitigation strategies. These techniques do not eliminate reward hacking entirely, as that would require a perfect reward model. However, they constrain reward hacking to manageable levels, letting us to capture the benefits of optimization while limiting its downsides. The strategies operate on different principles: some constrain how far the policy can drift, others make the reward signal more reliable, and others sidestep the optimization problem altogether.

Understanding these strategies is needed for implementing effective RLHF systems, and each one represents a different perspective on the same underlying problem: we need optimization to improve the model, but we need to prevent optimization from exploiting the proxy rather than improving the underlying quality.

The most basic and widely used mitigation is adding a penalty for deviating from the reference policy. The intuition is straightforward: if we know that the reward model is reliable near the reference distribution but unreliable far from it, we should penalize the policy for straying too far. Think of it as adding a rubber band between the optimizing policy and the reference policy. The band allows the policy to move toward higher reward, but it pulls back harder as the policy drifts further from the reference, preventing extreme exploration into unreliable territory.

Instead of maximizing raw reward, we optimize a modified objective that balances reward against divergence:

$$J(\theta) = \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta}[R_\phi(x, y)] - \beta\, D_{\text{KL}}(\pi_\theta \| \pi_{\text{ref}})$$

where:

• $J(\theta)$: the objective function we want to maximize
• $\theta$: the parameters of the policy network
• $\mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta}$: the expectation over prompts $x$ from the dataset and responses $y$ sampled from the current policy
• $R_\phi(x, y)$: the reward model score for response $y$ given prompt $x$
• $\beta$: a scalar coefficient controlling the strength of the KL penalty (larger $\beta$ means more conservative optimization)
• $D_{\text{KL}}(\pi_\theta \| \pi_{\text{ref}})$: the Kullback-Leibler divergence between the current policy and the reference policy, measuring how much the distribution of outputs has changed

The KL divergence term computes, on average over all possible outputs, how much more likely the current policy is to generate each output compared to the reference policy. When the policies are identical, the KL divergence is zero and there is no penalty. As the policies diverge, the KL divergence grows, imposing an increasing cost on exploration. This cost forces the optimization to weigh the reward benefit of moving into new territory against the penalty for diverging from the reference.

The coefficient $\beta$ controls the strength of the constraint and represents a necessary hyperparameter that you must tune carefully. Higher $\beta$ values keep the policy closer to the reference distribution, preventing extreme exploration into unreliable regions of the reward model but also limiting the potential gains from optimization. Lower $\beta$ values allow more aggressive optimization, potentially achieving higher rewards but with greater risk of reward hacking. The optimal choice of $\beta$ depends on the quality of the reward model, the desired level of improvement, and the acceptable level of risk.

The plot demonstrates how the KL penalty shapes the optimization objective. Without any penalty (the dark purple line with $\beta = 0$), the effective objective continues to increase as the policy drifts further from the reference. This provides no natural stopping point and inevitably leads to severe over-optimization. With increasing penalty strength (lighter colors), the effective objective reaches a peak and then declines. This creates a natural stopping point where the marginal benefit of additional reward no longer outweighs the penalty for divergence. The dots mark these peaks for each nonzero $\beta$ value, showing how stronger penalties lead to earlier optimal stopping points. We will explore the KL divergence penalty in detail in upcoming chapters on PPO and policy gradient methods, where we will see how it integrates into the full optimization loop.

Using an ensemble of multiple reward models reduces the impact of individual model errors. The point is that different reward models, trained on the same data but with different initializations or architectures, will have different weaknesses. If a policy exploits a specific weakness in one reward model, other members of the ensemble are unlikely to share that exact weakness. By aggregating across multiple models, we average out individual errors and create a more reliable signal.

Think of a reward model ensemble as a panel of judges rather than a single judge. An optimizer can exploit one judge's biases or blind spots, including idiosyncratic preferences. A panel of judges with different perspectives and different blind spots is much harder to game simultaneously: an answer that exploits one judge's bias is likely to fail with the others.

The ensemble reward is typically computed as the simple average across all models:

$$R_{\text{ensemble}}(x, y) = \frac{1}{K} \sum_{k=1}^{K} R_{\phi_k}(x, y)$$

where:

• $R_{\text{ensemble}}(x, y)$: the ensemble mean reward score for response $y$ to prompt $x$
• $K$: the number of reward models in the ensemble
• $R_{\phi_k}(x, y)$: the score predicted by the $k$-th reward model with parameters $\phi_k$

Alternatively, using the minimum across the ensemble provides a more conservative estimate that offers stronger protection against exploitation:

$$R_{\text{conservative}}(x, y) = \min_{k}\, R_{\phi_k}(x, y)$$

where:

• $R_{\text{conservative}}(x, y)$: the conservative reward estimate, taking the lowest score across all ensemble members
• $\min_{k}$: the minimum operator selecting the lowest score among all models

The conservative approach is particularly effective because it requires the policy to satisfy all reward models simultaneously. To achieve a high conservative reward, the policy must produce outputs that every model agrees are good. If even one model correctly identifies that an output is exploitative, the conservative reward will be low. This creates a much more reliable optimization target, though it may also be more pessimistic and harder to optimize against, since it discards useful information from models that correctly assign high scores.

The visualization confirms the benefits of ensemble methods through improved correlation with true quality. While individual models show significant scatter around the ideal prediction line (left panel), the ensemble mean tightens this scatter substantially (center panel). The correlation coefficient shown in each title quantifies this improvement. The conservative minimum (right panel) shows even less scatter above the ideal line. This reflects its tendency to filter out samples where any model predicts a low score. This approach is particularly effective at catching exploitative outputs that fool some but not all models in the ensemble.

Another sophisticated approach penalizes the policy for generating outputs where the reward model is uncertain about its predictions. The core idea is that if we can estimate the reward model's confidence, we can discourage the policy from exploring low-confidence regions where predictions are unreliable and exploitation is likely.

For ensemble methods, the disagreement between ensemble members provides a natural and interpretable uncertainty estimate. When all models in the ensemble agree on a score, we have high confidence that the prediction is reliable. When models disagree significantly, we have evidence that we are in a region where individual models have different weaknesses, and predictions should be treated with caution.

The uncertainty can be quantified as the standard deviation across ensemble predictions:

$$\sigma_R(x, y) = \sqrt{\frac{1}{K} \sum_{k=1}^{K} \left(R_{\phi_k}(x, y) - R_{\text{ensemble}}(x, y)\right)^2}$$

where:

• $\sigma_R(x, y)$: the uncertainty estimate, measured as the standard deviation of reward predictions across the ensemble
• $K$: the number of models in the ensemble
• $R_{\phi_k}(x, y)$: the score from the $k$-th model
• $R_{\text{ensemble}}(x, y)$: the mean score across the ensemble, as defined above

The modified objective then incorporates this uncertainty as a penalty term, discouraging the policy from generating outputs that cause the ensemble to disagree:

$$J(\theta) = \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta}\left[R_{\text{ensemble}}(x, y) - \lambda\, \sigma_R(x, y)\right]$$

where:

• $J(\theta)$: the uncertainty-penalized objective function
• $\lambda$: a hyperparameter controlling the strength of the uncertainty penalty (larger $\lambda$ means more conservative optimization)
• The other terms are as defined above

This uncertainty penalty discourages the policy from venturing into regions where reward predictions are unreliable. Even if a particular output might receive a high mean reward, if the models disagree significantly about that score, the uncertainty penalty will reduce the effective reward. The result is a form of pessimistic optimization: the policy prefers outputs that it can confidently identify as good over outputs that might be excellent or might be exploits.

The left panel shows the mean reward surface alone: optimization would drive the policy toward the upper-right corner where rewards are highest. However, the right panel reveals that after applying the uncertainty penalty, the optimal region shifts. The high-reward region in the corner is also high-uncertainty (where ensemble models disagree), so the penalized objective steers optimization toward a safer region where the ensemble confidently agrees on reasonable rewards. The green star marks this "safe optimum" that balances reward against prediction confidence.

Rather than training the reward model once and optimizing against it indefinitely, iterative approaches alternate between optimization and reward model refinement. This process keeps the reward model's training distribution aligned with the policy's output distribution, reducing the severity of distribution shift. The approach proceeds in cycles: optimize the policy against the current reward model, collect new preference data from the optimized policy, update the reward model with the new data, then repeat.

As the policy learns to produce new kinds of outputs, those outputs are evaluated by humans and added to the reward model's training set. The reward model then learns to evaluate these new outputs accurately, closing the gap that the policy might otherwise exploit. Think of this as updating the map as the traveler explores new territory: rather than letting the traveler wander off into unmapped terrain where the map's errors can be catastrophic, you update the map to cover the new ground before the traveler moves further.

However, iterative approaches require ongoing human annotation effort, which can be expensive and time-consuming. Each iteration requires collecting new human preferences, which may slow down your development cycle. Additionally, there is a risk of the reward model learning to track the policy's outputs rather than learning stable quality criteria, potentially leading to a different kind of optimization failure.

The visualization shows how iterative updates maintain closer alignment between distributions. In early iterations, the policy distribution (gray ellipse) may drift ahead of the reward model's training data (colored ellipse), creating a distribution gap where the reward model must extrapolate. Through iterative updates, new data is collected from the current policy and added to reward model training, expanding the training distribution to cover where the policy is currently generating outputs. The "Distribution Gap" metric quantifies this alignment across iterations, decreasing from 1.75 to 0.22 as the reward model catches up with the policy.

A simpler and more conservative alternative to policy optimization is best-of-N sampling. Instead of modifying the policy weights through gradient-based optimization, we generate N candidate responses from the base policy and select the one with the highest reward score. This approach achieves some of the benefits of optimization without the risks of aggressive policy modification.

Best-of-N sampling is less susceptible to extreme reward hacking because it only samples from the base policy distribution. The policy itself is never modified, so it cannot learn to produce outputs that exploit reward model weaknesses. Each candidate response comes from the same distribution as the reference policy. This keeps all candidates fall within the region where the reward model was trained. The reward model is only used to select among these candidates, not to guide gradient-based optimization toward potentially problematic regions.

However, best-of-N sampling has significant limitations. It is computationally expensive, requiring N forward passes per query, which can make it impractical for high-throughput applications. It also cannot achieve as much improvement as direct optimization when the optimization target is well-specified: direct optimization can make systematic changes to the policy that improve performance across the board, while best-of-N only selects among the outputs the base policy was already capable of producing. Best-of-N is best understood as a baseline and a diagnostic tool rather than a primary alignment strategy.

Constitutional AI methods represent a fundamentally different approach to the reward hacking problem. Instead of relying primarily on learned reward models that capture statistical patterns from human preferences, these methods use language model self-evaluation against explicit, written principles. The model evaluates its own outputs by checking whether they adhere to criteria like "Be honest," "Do not assist with harmful tasks," or "Acknowledge uncertainty when appropriate."

This approach is less susceptible to reward hacking because the evaluation criteria are explicit rather than learned from statistical patterns, which means there are fewer hidden biases in the evaluation. Self-evaluation uses the model's own understanding of language and concepts rather than a separate neural network that must extrapolate across distribution shift. The principles can be directly inspected and modified by humans. This provides a more interpretable form of alignment. When the model evaluates its own output against a written principle, the evaluation process is transparent in a way that a neural reward model's predictions are not.

However, constitutional approaches have their own limitations. The model's ability to correctly interpret and apply abstract principles depends on its understanding of language and ethics, which may itself be imperfect or manipulable. A model might learn to generate outputs that satisfy its self-evaluation while still failing to satisfy the spirit of the principles. This is, in a sense, reward hacking at a higher level of abstraction: instead of exploiting a neural reward model, the model might learn to exploit its own self-evaluation procedure.