Reward Modeling › Summary

This chapter covered the complete pipeline for building reward models that learn to predict human preferences and serve as the optimization target for RLHF.

Architecture: Reward models add a value head to pretrained transformers, projecting the final token's hidden state to a scalar reward. Using the same architecture family as the policy model ensures compatible representations. The value head is typically a single linear layer, though more complex heads can be used. The final token's position is preferred because causal attention accumulates information from the entire sequence.

Loss function: The Bradley-Terry preference loss $\mathcal{L} = -\log \sigma(r_w - r_l)$ maximizes the probability of correctly ranking preference pairs. Gradients naturally emphasize uncertain or incorrect predictions. The loss is shift-invariant: only reward differences matter, not absolute values. Margin-based variants encourage larger score separations but can hurt calibration.

Training: Conservative hyperparameters preserve pretrained knowledge while learning preference structure. Early stopping, gradient clipping, label smoothing, and appropriate regularization prevent overfitting to limited preference data. Post-training reward normalization ensures the reward signal is in a consistent range for downstream RL optimization.

Evaluation: Pairwise accuracy measures ranking performance, with human agreement giving an upper bound. Calibration testing, agreement with human evaluators, and adversarial probing for biases like length preference or style over substance are all needed components of complete evaluation. The evaluation must address in-distribution accuracy, out-of-distribution generalization, and susceptibility to exploitation.

Limitations: Reward models are proxies for human preferences and are bounded by annotation quality, susceptible to distribution shift, and vulnerable to reward hacking. These limitations motivate both the KL penalty in RLHF and alternative preference learning methods. Understanding these failure modes is as important as understanding the training procedure.

The reward model is the necessary interface between human preferences and policy optimization. In the following chapters, we'll examine how reward hacking can undermine this proxy relationship, and then explore how policy gradient methods and PPO use reward signals to improve language models.