Reward Hacking › Reward Hacking

In the previous chapter, we built reward models that predict human preferences. These models assign scalar scores to language model outputs, with the goal of guiding optimization toward responses that humans would prefer. The promise is appealing: collect some human feedback, train a model to predict it, and then use that model as an automatic judge to steer the language model toward better behavior. In principle, this lets us scale alignment beyond what direct human supervision could provide. In practice, a necessary question emerges almost immediately: what happens when a language model discovers ways to achieve high reward scores without producing the outputs humans intended?

This phenomenon is known as reward hacking, and it is a major challenge in aligning language models with human intentions. Think of the reward model as a student who was given a set of sample problems before an exam. Within those samples, the student learned effective strategies. But when the exam includes questions far outside the practice set, those strategies break down, and the student may resort to guessing or copying superficial patterns without understanding them. The language model, optimizing against a reward model, faces exactly this situation: the reward model is reliable near the training distribution but increasingly unreliable as optimization pushes the policy into novel territory.

The reward model is an imperfect proxy for what humans want. When we optimize aggressively against this proxy, the language model can find unexpected strategies that exploit gaps between the reward signal and human preferences. These exploitations are not the result of a malicious or deceptive language model. They emerge from the optimization process, which searches for patterns that lead to higher scores, regardless of whether those patterns correspond to quality. Optimization is agnostic about the relationship between the objective it maximizes and the outcome we care about. It simply finds the highest values it can reach, exploiting any discrepancy between proxy and truth along the way.

Understanding reward hacking is needed before we proceed to the policy optimization methods in upcoming chapters, because those techniques only work well when we account for the limitations of learned reward signals. The mitigation strategies we develop in this chapter, especially the KL divergence penalty, will recur as central components of PPO and other alignment algorithms. This chapter gives you the conceptual foundation to understand that these constraints exist, why they are necessary, and what failure mode they prevent.