Reward Hacking › The Fundamental Problem

Reward hacking occurs when an agent optimizes for a reward signal in ways that achieve high scores without fulfilling the intended objective. In the context of language models, this means generating text that receives high reward model scores while failing to be helpful, accurate, or aligned with human values. The phenomenon does not require the model to have any explicit intent to deceive. It emerges from the optimization process, which searches for patterns that lead to higher scores, regardless of whether those patterns correspond to quality.

Think of the reward model as a map and the language model as a traveler trying to reach the highest point on that map. If the map accurately represents the terrain, the traveler will reach a real peak. But if the map has errors, especially in unexplored regions, the traveler might follow the map to a location that appears high on paper while standing in a valley in reality. The optimization process cannot distinguish between "this region scores high because it is good" and "this region scores high because the reward model made an error here." Both look equally attractive from the optimizer's perspective.

The root cause of reward hacking lies in a basic distinction: the difference between the true human preference function and our learned approximation of it. To understand this distinction clearly, consider the goal. Ideally, we want a language model that generates helpful, accurate responses suited to the user's request. If we could somehow access a perfect oracle that encodes all human values and preferences, we would optimize directly against that oracle. Let $R^*(x, y)$ denote this idealized reward function, which perfectly captures what humans would prefer for any given prompt $x$ and response $y$. This function represents the ground truth of human preference, accounting for all the nuance, context-dependence, and complexity of what makes one response better than another.

What we have, however, is something far more limited: $R_\phi(x, y)$, a neural network trained on a finite dataset of human comparisons. This learned reward model represents our best attempt to approximate the true preference function using the data and computational resources available to us. The relationship between these two functions can be expressed as:

where:

• $R_\phi(x, y)$: the learned reward model's prediction for prompt $x$ and response $y$
• $R^*(x, y)$: the true human preference (the idealized reward)
• $\epsilon(x, y)$: the approximation error between the learned model and true preference
• $x$: the input prompt
• $y$: the generated response

This decomposition reveals the core of the problem. The error term $\epsilon(x, y)$ is not simply random noise uniformly distributed across all inputs and outputs. Rather, it has systematic structure that depends on several factors: the distribution of examples in the training data, the architectural choices made when building the reward model, the biases and inconsistencies present in human annotations, and the inherent limitations of representing a complex, multidimensional preference function with a neural network. Some regions of the output space will have small errors because they are well-represented in the training data and the patterns there are consistent. Other regions will have large errors because the reward model has seen few similar examples or because human annotators disagreed about preferences in those areas.

When we optimize a policy $\pi_\theta$ to maximize $R_\phi$, we are essentially instructing the optimization process to find outputs that score highly according to the learned reward model. The optimization algorithm has no way of knowing which high-scoring outputs are good (where $R_\phi$ closely approximates $R^*$) versus which high-scoring outputs are exploiting errors in the approximation (where $\epsilon(x, y)$ is large and positive). From the optimizer's perspective, both look equally valuable. This creates a systematic pressure toward discovering and exploiting regions where the reward model overestimates quality, achieving high reward scores despite low true preference.

The visualization illustrates how the error term varies across output space. In the left panel, regions near the center (where training data is concentrated) show low approximation error, while peripheral regions show high error. The right panel shows the corresponding training data density. Optimization pressure naturally drives the policy toward high-error regions where the reward model overestimates quality, because those are precisely the regions where exploitation is possible. The reward model cannot warn us when it is extrapolating unreliably; it simply produces confident-looking predictions regardless.