Reward Hacking › Summary

Reward hacking represents a basic challenge in aligning language models with human preferences. When we optimize against a learned reward model, the optimization process can discover ways to achieve high scores without creating the outputs humans want. This happens because the reward model is an imperfect proxy for true human preferences, and optimization is agnostic about this imperfection: it exploits errors in the proxy just as readily as it captures useful quality signals.

The key mechanisms driving reward hacking are:

• Proxy-truth mismatch: The learned reward model $R_\phi$ approximates but does not equal the true human preference $R^*$, and the error term $\epsilon(x, y) = R_\phi - R^*$ has exploitable structure.
• Distribution shift: As optimization pushes the policy away from the reference distribution, the reward model must extrapolate into unfamiliar territory where its predictions become unreliable.
• Over-optimization: Continued optimization past the quality peak compounds small errors into large ones, following the scaling law $R^* \approx \alpha\sqrt{D_{\text{KL}}} - \beta D_{\text{KL}}$.
• Behavioral exploits: Specific patterns like length inflation, sycophancy, format gaming, and question-echoing emerge from spurious correlations learned by the reward model.

Mitigation strategies include KL divergence penalties to constrain policy drift, reward model ensembles to reduce individual model vulnerabilities, uncertainty estimation to discourage exploration into unreliable regions, iterative reward model updates to maintain alignment between training and policy distributions, and best-of-N sampling as a conservative alternative to direct optimization. No single strategy is sufficient; in practice, effective RLHF systems combine several of these approaches.

As we move into the upcoming chapters on policy gradient methods and PPO, understanding reward hacking will be important. The techniques for optimizing language model policies are designed with these failure modes in mind, incorporating constraints and regularization specifically to prevent the optimization pathologies we have examined here. The KL divergence penalty, in particular, will play a central role in making RLHF optimization stable and effective despite the inherent limitations of learned reward models.