Reward Modeling › Reward Modeling

In the previous chapter, we explored the Bradley-Terry model and how it provides a probabilistic framework for converting pairwise human preferences into a consistent scoring system. Now we turn to the practical question: how do we build a neural network that learns to predict these preferences at scale?

A reward model is a neural network that takes a prompt and response as input and outputs a scalar score showing how "good" that response is according to human preferences. This model is a proxy for human judgment. This allows us to provide dense feedback signals during reinforcement learning without requiring a human to evaluate every generated response. The reward model sits at the heart of RLHF: it translates sparse, noisy human preferences into a continuous signal that can guide policy optimization. Think of the reward model as a learned critic that has internalized the aesthetic sensibilities of your annotator pool. Once trained, it can evaluate thousands of responses per second, standing in for the human evaluators who could only score a small fraction of that volume.

Building an effective reward model requires careful consideration of architecture choices, loss functions, and evaluation methods. The model must generalize from a limited set of human comparisons to accurately score responses it has never seen before, including responses generated by future versions of the language model being trained. This generalization challenge is more subtle than typical supervised learning. We are not just asking the model to recognize patterns in text. We are asking it to internalize a subjective, context-dependent value system from a collection of binary comparison labels, then extrapolate that value system to an unbounded space of possible outputs. The better the model generalizes, the more reliably the downstream RL process will produce helpful behavior rather than exploiting statistical artifacts.

Consider the practical scale of the challenge. A human annotator reviewing responses might evaluate fifty to one hundred pairs per hour. A language model undergoing RLHF training might generate millions of candidate responses across the training run, each requiring a quality signal. Without an automated proxy, RLHF would be computationally intractable. The reward model bridges this gap, but only if it has learned a sufficiently faithful representation of human values. Every imperfection in the reward model becomes an adversarial opportunity for the policy optimizer, which will relentlessly probe for responses that exploit whatever inconsistencies exist in the learned scoring function.

The degree to which reward models succeed in this role explains much of the variation in quality between different RLHF-trained systems. Two language models trained with the same base architecture but different reward models can exhibit dramatically different behavior. A reward model that overweights surface stylistic features (confident tone, formatted lists, certain vocabulary) will steer the policy toward responses that look helpful without necessarily being helpful. A reward model that captures the substance of human preferences will instead favor accurate, informative responses suited to the request. Understanding reward-model construction and training, including how to evaluate the result, therefore gives you direct insight into why aligned language models behave the way they do.

This chapter covers the complete pipeline from architecture design through training and evaluation. We begin with the architectural choices that determine how a pretrained transformer is adapted into an evaluative model. We then derive the preference loss function from first principles and examine its mathematical properties. The practical training section covers data preparation and optimization, followed by the regularization choices that control overfitting. We cover evaluation methodology in depth, including the metrics that matter and the failure modes to watch for. A numerical worked example makes the loss computation concrete. The code section implements a complete reward model from scratch. We close by examining the basic limitations that make reward modeling one of the most active research areas in alignment.