Reward Modeling › Limitations and Impact

Reward modeling is a powerful technique but comes with significant challenges that you must understand before deploying it in production RLHF systems. These limitations are not merely theoretical: they have caused real quality problems in deployed language models and have motivated major lines of research in the alignment community. Understanding where reward models can fail, and why, is as important as understanding how to build them.

The basic limitation of reward models is that they are proxies for human preferences, not perfect representations. This is the key tension at the heart of RLHF: we want to optimize for human values, but the RL optimizer sees only the reward model's approximation of those values. Every error in the approximation becomes an opportunity for the optimizer. The model learns from a finite set of comparisons made by a specific group of annotators under particular conditions. It cannot generalize perfectly to all possible responses or capture the full complexity of human values. When the policy model optimizes against this learned reward, it may find responses that score highly according to the proxy but do not satisfy human preferences.

Think of the reward model as a map and human preferences as the territory. A map can be useful for navigation even when it is not perfectly accurate. But under severe optimization pressure, the map's imperfections start to matter: a path that looks short on the map may be impassable in reality. Similarly, a policy that is trained for just a few RL steps will likely improve. A policy that is trained for many RL steps under high optimization pressure will start exploiting the gaps between the map and the territory. This phenomenon, known as reward hacking or Goodhart's Law in action, becomes more severe as optimization pressure increases. We'll explore this challenge in depth in the next chapter.

Reward model quality is bounded by the quality of the underlying preference data. Human annotators disagree, make mistakes, and have biases. If 70% of annotators prefer response A over B, the "correct" label is somewhat arbitrary. The reward model learns from these noisy, inconsistent signals, and this uncertainty propagates into the learned reward function.

Different annotator pools may have systematically different preferences based on cultural background, expertise, or task understanding. A reward model trained on one population may not generalize to another. This is particularly concerning when the intended users of the deployed model are demographically or culturally different from the annotator pool. The reward model may have learned preferences that are appropriate for one group but misaligned with another.

The annotation interface and instructions also matter. Subtle differences in how the comparison task is framed can lead to systematic biases in the annotations. If annotators are asked to select "the better response" without further guidance, some may focus on helpfulness, others on tone, others on accuracy, and still others on length. The resulting reward model will reflect this mixture of criteria, which may not align with any single coherent value system. Clear annotation guidelines that specify exactly what "better" means for the intended application reduce this problem but cannot eliminate it entirely.

During RLHF training, the policy model generates responses that may differ substantially from those in the reward model's training set. The reward model must extrapolate to these out-of-distribution samples, and its predictions become less reliable as the policy's outputs diverge from the training distribution. This creates a feedback loop: the policy learns to generate responses that score well according to the reward model's potentially incorrect extrapolations, which can lead to degraded actual quality even as measured rewards increase.

The KL divergence penalty used in most RLHF implementations (we'll cover this in detail in the PPO chapter) is designed to limit this distribution shift. By penalizing the policy for deviating too far from the SFT initialization, the KL penalty keeps the policy's outputs in a region where the reward model's extrapolations are more likely to be reliable. However, this is a blunt instrument: it slows policy improvement overall, not just in areas where the reward model is unreliable.

Research on reward model ensembles addresses this limitation by using multiple reward models trained on different subsets of the data or with different architectures. When the ensemble members disagree on a response, that response is likely out-of-distribution for at least some models, and the disagreement can be used as an uncertainty signal. Conservative RL algorithms can then be penalized for exploring uncertain regions, focusing optimization on areas where the reward is reliably estimated.

Training reward models requires significant computational resources, particularly when using large base models to ensure the reward model has sufficient language understanding. A 70B parameter reward model requires more GPU memory and compute than most teams can afford for a component that serves only as an auxiliary signal. During RL training, the reward model must evaluate every generated response, adding substantial inference costs to an already expensive training procedure.

There are several strategies for managing these costs. First, use a reward model that is smaller than the policy, accepting some quality loss for throughput. Second, apply reward model distillation: train a large, high-quality reward model once, then distill it into a smaller model for use during RL training. Third, cache reward model evaluations when the policy generates the same or similar responses repeatedly. Fourth, use reward model parallelism: if you have multiple GPUs, run the reward model on a dedicated subset while the policy runs on others.

The trend toward larger models creates an escalating cost dynamic. As policies grow larger, more capable reward models are needed to evaluate them accurately, which in turn increases inference costs. This scaling challenge is one of the motivations for DPO and other reward-model-free approaches to preference learning.

Beyond the proxy problem, reward models are vulnerable to specification gaming: the policy may find valid-seeming ways to achieve high rewards that nonetheless violate the intent of the human preferences. Common examples include verbosity (padding responses with filler content to appear thorough), hedging (adding excessive caveats to avoid making claims that could be wrong), sycophancy (agreeing with the user even when the user is incorrect), and false confidence (asserting uncertain facts with unwarranted certainty).

These behaviors can be hard to detect from reward model scores alone because the reward model was trained on the same distribution that generated the problematic preference labels. If human annotators preferred verbose responses during training, the reward model will score verbose responses highly, and the policy will produce verbose responses. The problem only becomes visible when humans assess the RL-trained policy and notice that it produces unnecessarily long, formulaic answers. By that point, the policy may already be deeply trained on the problematic reward signal, and retraining requires collecting new annotations specifically targeted at the identified bias.

The best defense against specification gaming is proactive adversarial assessment of the reward model before RL training begins. Construct response pairs specifically designed to test hypothesized weaknesses, collect human judgments on those pairs, and verify that the reward model's scores align with human judgments. This kind of red-teaming is time-consuming but far cheaper than discovering specification gaming problems after a full RL training run.

Despite these limitations, reward models have enabled major advances in language model alignment. They provide the important bridge between sparse human feedback and dense training signals. The InstructGPT system that powers ChatGPT, Anthropic's Claude models, and many open-source chat models all rely on reward models as a core component. The quality improvements these systems demonstrate over base language models are directly attributable to the reward model's ability to encode human preferences and feed them into the RL training loop.

The reward modeling approach has also influenced research directions, spurring work on direct preference optimization (DPO) methods that eliminate the need for explicit reward models, as well as techniques for reward model ensembles, uncertainty quantification, and reliable optimization. Understanding reward modeling deeply is needed for grasping both current RLHF systems and the alternatives being developed to address its limitations. The basic challenge of learning human values from comparisons, and building a representation of those values reliable enough to withstand adversarial optimization, remains one of the central open problems in the field.