Reward Hacking › Distribution Shift

Distribution shift is a primary driver of reward hacking, and understanding it deeply is needed for appreciating why reward models fail under optimization pressure. The core insight is that the reward model is trained on a specific distribution of outputs: those generated by the initial policy $\pi_{\text{ref}}$, typically a supervised fine-tuned model. The reward model learns to evaluate and rank responses within this distribution effectively, distinguishing better responses from worse ones among the kinds of outputs that the reference policy produces. However, as optimization progresses, the current policy $\pi_\theta$ drifts away from $\pi_{\text{ref}}$, generating outputs that lie increasingly outside the training distribution of the reward model. In these unfamiliar regions, the reward model's predictions become unreliable.

Think of distribution shift as a chef who trained on French cuisine suddenly being asked to evaluate molecular gastronomy. Within French cuisine, the chef has deep expertise and can reliably distinguish excellent from mediocre. But molecular gastronomy uses different techniques, different textures, and different flavor profiles. The chef can still form opinions, but those opinions are based on extrapolating from French culinary principles rather than expertise in the new domain. Similarly, the reward model extrapolates from its training distribution into new territory, and those extrapolations become increasingly unreliable the further the policy drifts.

During reward model training, we collect preferences over responses generated by a supervised fine-tuned model. This creates a specific distribution of outputs shaped by what the base model can do, how it tends to respond, and where it fails. Responses from this model might be helpful but occasionally verbose, accurate on common topics but uncertain on obscure ones, polite and professional in tone. The reward model learns to rank responses within this distribution effectively, picking up on patterns that distinguish better responses from worse ones among the kinds of outputs the reference policy tends to produce.

But during RLHF optimization, something fundamentally changes. The policy is no longer constrained to produce outputs like the reference model. Instead, it actively explores new regions of output space in search of higher reward scores. As optimization progresses, the policy learns to produce outputs that are increasingly different from anything the reward model saw during training. Perhaps it discovers that certain unusual phrasings score higher, or that particular structural patterns receive higher rewards. These discoveries push the policy further and further from the reference distribution, into territory where the reward model must rely on extrapolation rather than interpolation.

The problem is that the reward model cannot recognize when it is extrapolating. It has no explicit representation of its training distribution boundary or uncertainty indicator. It simply applies its learned parameters to whatever input it receives and produces a scalar score. In well-represented regions, this score is meaningful. In poorly represented regions, this score reflects the extension of learned patterns into unfamiliar territory, which may bear little relationship to actual human preferences.

The following simulation creates a 2D latent space to visualize how the policy distribution drifts away from the reward model's training distribution during optimization.

The visualization reveals the progressive nature of distribution shift during optimization. In the reliable region near the reference policy distribution (shown in blue and highlighted by the shaded circle), the reward model's predictions correlate well with true human preferences because this is where it was trained to operate. The orange distribution shows the policy after light optimization: still overlapping with the training distribution but beginning to drift. The red distribution shows the result of heavy optimization: the policy has converged to a narrow region far from the training data, where reward model scores may be high but predictions are increasingly divorced from actual quality. The dashed ellipses mark the approximate boundaries of each distribution, showing how the spread narrows as the policy specializes in exploiting specific reward model patterns.

Neural networks, including reward models, are notoriously poor at extrapolation. This limitation is basic to how these models learn and generalize. Within the training distribution, the reward model has seen many examples of responses with various qualities and has learned meaningful patterns about what distinguishes preferable responses from less preferable ones. It can interpolate effectively between seen examples, making reasonable predictions about new responses that fall within the distribution of its training data.

Outside this distribution, however, the reward model's predictions are based on extrapolation from limited data. The model must extend its learned patterns into regions it has never seen, and there is no guarantee that the patterns that hold within the training distribution continue to hold outside it. A feature that consistently correlated with quality in training (like using technical terminology appropriately) might correlate with nonsense outside the training distribution (like stringing together technical-sounding words without coherent meaning). The optimization process is particularly effective at finding these extrapolation failures because it systematically searches for inputs that maximize the reward model's predictions, regardless of whether those predictions are reliable.

The mathematical relationship between distribution shift and prediction reliability can be expressed through the variance of the reward model's predictions. For outputs near the training distribution, the reward model has low predictive variance and high confidence. For out-of-distribution outputs, the model has high uncertainty that it cannot reliably quantify. The reward model typically does not know that it is extrapolating: it produces confident predictions regardless of whether those predictions are trustworthy.

We can decompose the total predictive variance into two components:

$$\text{Var}[R_\phi(x, y)] \approx \sigma^2_{\text{base}} + \sigma^2_{\text{dist}}(d(y, \mathcal{D}))$$

where:

• $\text{Var}[R_\phi(x, y)]$: the predictive variance (uncertainty) of the reward model
• $\sigma^2_{\text{base}}$: the baseline uncertainty (aleatoric) inherent in the data itself, which reflects annotator disagreement
• $\sigma^2_{\text{dist}}$: the epistemic uncertainty component that grows as inputs move away from training data
• $d(y, \mathcal{D})$: a metric measuring the distance between output $y$ and the training distribution $\mathcal{D}$

The first term, $\sigma^2_{\text{base}}$, is irreducible. It reflects the fact that human preferences are inherently noisy and sometimes inconsistent. Even with infinite training data, different annotators would disagree on some comparisons, and the same annotator might rate a pair differently on different days. The second term, $\sigma^2_{\text{dist}}$, is the concerning one for reward hacking. It grows as the policy's outputs move away from the training distribution. This reflects the model's increasing lack of reliable information about preferences in unfamiliar regions. This distance-dependent term represents the growing unreliability of reward model predictions under optimization pressure.