Reward Hacking › Over-Optimization

Over-optimization, sometimes called reward hacking through optimization pressure, occurs when we optimize too aggressively against the reward model. This phenomenon is distinct from the individual errors we discussed earlier. Even if each individual reward model error is small, sufficient optimization pressure can find and exploit these errors, amplifying them into significant quality degradation. The key insight is that optimization is a systematic search process that actively seeks out weaknesses in the objective function, and it has essentially unlimited patience to do so.

Think of over-optimization as a process of progressive exploitation. In the early stages, the policy finds the legitimate improvements: patterns the reward model correctly identifies as quality signals. These produce real gains. In the middle stages, the policy finds the easy exploits: surface features that correlate with quality in the training data but can be produced without the underlying quality. These produce inflated scores with no improvement, or even a slight decrease, in true quality. In the late stages, the policy finds the deep exploits: subtle patterns in the reward model's learned representations that allow misaligned outputs to receive high scores. At this point, true quality is actively declining while proxy reward continues to rise.

Research has demonstrated a consistent and reproducible pattern in RLHF optimization: as optimization against a reward model increases, true quality (as measured by held-out human evaluations) initially improves, reaches a peak, and then degrades. This characteristic curve defines the over-optimization problem and determines when policy training should stop.

In the early stages of optimization, the policy improves. It picks up on patterns that the reward model has correctly identified as markers of quality: providing helpful, accurate information that responds to your intent. During this phase, the reward model is an effective guide, steering the policy toward better responses. True quality improves alongside the proxy reward score.

As optimization continues, however, the policy begins to exhaust the "easy" improvements and starts finding more subtle patterns. Some of these patterns continue to improve quality, but increasingly, the policy discovers patterns that exploit quirks in the reward model rather than reflecting true preference. The policy might learn to use certain phrases that the reward model associates with quality, even in contexts where those phrases are inappropriate. It might adopt formatting conventions that score well but reduce clarity. Each of these exploitations provides a small boost to the proxy reward while giving little or no benefit, or even harm, to true quality.

Eventually, the degradation from exploitation outweighs the benefits from valid improvements. True quality begins to fall even as proxy reward continues to rise. The policy has learned to game the reward model, creating outputs that look good to the proxy but would disappoint human evaluators.

We can model this relationship by defining functions for true_quality_curve and proxy_reward_curve, then plotting them as the KL divergence increases. The KL divergence is a measure of how far the policy has drifted from its starting point, which corresponds to the intensity of optimization.

The divergence between proxy reward and true quality represents the core over-optimization problem visualized directly. The blue line shows the target: true quality as measured by held-out human evaluations. The red dashed line shows what the optimization process sees: the proxy reward from the reward model. The green vertical line marks the optimal stopping point, where true quality reaches its peak. Beyond this point, continued optimization makes things worse, not better. The red shaded region highlights the "over-optimization gap," which grows as optimization continues past the peak.

Empirical research has characterized how over-optimization scales with various factors. This provides quantitative insight into this phenomenon. The relationship between true quality $R^*$ and the optimization divergence $D_{\text{KL}}$ follows approximate scaling laws that have been validated across multiple experiments and model scales. Before presenting the formula, note that this relationship captures two competing forces: the improvements from optimization (captured by a growing term) and the degradation from over-exploitation (captured by a linear term).

The empirical relationship takes the form:

$$R^* \approx \alpha \sqrt{D_{\text{KL}}} - \beta D_{\text{KL}}$$

where:

• $R^*$: the true quality of the generated response, as measured by a held-out gold reward model or human evaluation
• $\alpha$: a positive coefficient capturing the initial improvement rate, which depends on reward model quality and training data size
• $\beta$: a positive coefficient capturing the rate of degradation from over-optimization, which also depends on reward model quality
• $D_{\text{KL}}$: the KL divergence $D_{\text{KL}}(\pi_\theta \| \pi_{\text{ref}})$ measuring how far the optimized policy has drifted from the reference policy

This functional form captures the needed dynamics of over-optimization. The first term, $\alpha \sqrt{D_{\text{KL}}}$, represents the beneficial effects of optimization. As the policy moves away from its starting point, it learns improvements that increase true quality. The square root dependence indicates diminishing returns: the easiest improvements are found first, and each subsequent improvement requires more optimization effort. The second term, $\beta D_{\text{KL}}$, represents the harmful effects of over-optimization. As the policy drifts further from the training distribution, reward model errors accumulate linearly with distance. The linear dependence in the degradation term (versus square root in the improvement term) ensures that degradation eventually dominates, no matter how good the reward model is.

We can find the optimal stopping point analytically by taking the derivative of $R^*$ with respect to $D_{\text{KL}}$ and setting it to zero:

$$\begin{aligned} \frac{dR^*}{dD_{\text{KL}}} &= \frac{\alpha}{2\sqrt{D_{\text{KL}}}} - \beta = 0 \\ D_{\text{KL}}^* &= \left(\frac{\alpha}{2\beta}\right)^2 \end{aligned}$$

This optimal divergence $D_{\text{KL}}^*$ tells us exactly how much optimization we can do before quality starts to degrade. A larger ratio $\alpha / \beta$ (achieved by better reward models with larger $\alpha$ and smaller $\beta$) allows for more optimization before the peak. The peak quality itself is:

$$R^*_{\text{max}} = \alpha \sqrt{D_{\text{KL}}^*} - \beta D_{\text{KL}}^* = \frac{\alpha^2}{4\beta}$$

This formula reveals that peak quality scales quadratically with $\alpha$ and inversely with $\beta$. Doubling the quality of your reward model (increasing $\alpha$ by 2x and decreasing $\beta$ by 2x) would quadruple the achievable peak quality. This explains why reward model quality is so important for RLHF success: improvements to the reward model have compounding benefits.

The coefficients $\alpha$ and $\beta$ depend on reward model quality, training data size, and model capacity. Larger reward models with more training data have larger $\alpha$ (more initial benefit from optimization) and smaller $\beta$ (slower degradation as the policy drifts), which pushes the optimal stopping point further out and allows for more optimization before quality degrades. However, over-optimization eventually occurs regardless of reward model quality.

The visualization shows how different reward model quality settings affect the over-optimization curve. Poor reward models reach their peak early and decline rapidly. This provides only a small window of beneficial optimization. Excellent reward models allow substantially more optimization before quality degrades, with higher peak quality and a gentler decline. The dots mark the optimal stopping point for each configuration. Even the best reward model eventually suffers from over-optimization.

Over-optimization is a basic consequence of optimizing against a proxy objective, not an implementation problem that better engineering could solve. Several factors conspire to make it unavoidable:

Finite reward model capacity. The reward model has limited capacity to represent the full complexity of human preferences. Human preferences are fine-grained, context-dependent, and sometimes contradictory. No finite neural network can capture this complexity perfectly. Optimization finds edge cases where this limited representation fails.

Training data coverage. No matter how much preference data we collect, some regions of output space will remain uncovered. The space of possible text outputs is large, and we can only sample a tiny fraction of it for human evaluation. The policy can learn to occupy regions that were never represented in training, where the reward model must extrapolate from distant examples.

Annotator inconsistency. Human preferences are noisy and inconsistent. The same person might rate the same comparison differently on different days, and different people often disagree about which response is better. The reward model learns an average that individual annotators might disagree with, and optimization can exploit these disagreements.

Distribution shift compounds errors. As the policy drifts from the training distribution, reward model errors compound. A small error that was harmless in-distribution can become a large error when the model extrapolates far from its training data. The optimization process actively seeks out these compounding errors, following gradients toward regions where the reward model's extrapolations are most favorable, regardless of whether those extrapolations are accurate.

Each of these factors is structural, not accidental. We can reduce their severity through better data collection, better model architecture, and better training procedures, but we cannot eliminate them entirely. This is why mitigation strategies, rather than elimination strategies, are the practical approach to reward hacking.