Reward Modeling › Reward Model Evaluation

Evaluating reward models is important but challenging. Unlike language models where we can measure perplexity on held-out text, reward models are evaluated on their ability to predict human preferences. The evaluation must assess whether the learned reward function captures the underlying preference structure, not just memorizes the training comparisons.

Think of reward model evaluation as answering three distinct questions simultaneously. First, does the model correctly rank responses on held-out examples from the same distribution as the training data? Second, does it correctly rank responses from distributions it has not seen, including responses generated by the policy during RL training? Third, are there systematic biases or shortcuts in the model's scoring that could be exploited during RL optimization? A model can pass the first test while failing the second and third, which is the evaluation trap that allows reward hacking to occur. A thorough evaluation probes all three dimensions.

The difficulty of complete evaluation is one of the basic challenges in reward modeling research. We typically only have access to held-out human comparison data from the same annotation protocol used for training, which addresses only the first question. Measuring out-of-distribution generalization requires collecting new annotations for distribution-shifted samples, which is expensive. Detecting exploitable biases requires adversarial testing: constructing responses specifically designed to exploit suspected weaknesses and checking whether the reward model scores them inappropriately. This kind of adversarial red-teaming is resource-intensive but invaluable before deploying a reward model in RL training.

Pairwise accuracy is the most direct evaluation. On a held-out set of preference pairs, we measure how often the reward model assigns a higher score to the human-preferred response. This metric directly measures the model's ability to perform its intended function: distinguishing better responses from worse ones.

$$\text{Accuracy} = \frac{1}{N_{\text{test}}} \sum_{i=1}^{N_{\text{test}}} \mathbf{1}[r(x^{(i)}, y_w^{(i)}) > r(x^{(i)}, y_l^{(i)})]$$

where:

• $\text{Accuracy}$: the fraction of correctly ranked pairs
• $N_{\text{test}}$: the total number of examples in the test set
• $\sum_{i=1}^{N_{\text{test}}}$: summation over all test examples
• $x^{(i)}$: the prompt for the $i$-th test example
• $y_w^{(i)}, y_l^{(i)}$: the preferred and rejected responses for the $i$-th test example
• $\mathbf{1}[\cdot]$: the indicator function, evaluating to 1 if the condition is true and 0 otherwise
• $r(x^{(i)}, \dots)$: the predicted reward for the $i$-th pair's responses

The indicator function $\mathbf{1}[\cdot]$ returns 1 when the condition inside is true (the chosen response receives a higher reward) and 0 otherwise. Summing these indicators and dividing by the total count gives us the proportion of correctly ranked pairs.

A random model achieves 50% accuracy, while human inter-annotator agreement typically ranges from 65-80% depending on the task difficulty. A good reward model should approach but not necessarily exceed human agreement levels, since disagreements in the training data cap achievable performance. If humans themselves only agree 75% of the time on which response is better, we cannot expect the model to exceed this ceiling. In fact, a model achieving much higher accuracy might be exploiting artifacts in the data rather than capturing preferences. The ceiling imposed by inter-annotator agreement is not a failure of reward modeling but a reflection of ambiguity in human values, and reward models that appear to exceed this ceiling should be scrutinized carefully for data leakage or shortcut learning.

Calibration measures whether the model's confidence matches its accuracy. If the model predicts a preference probability of 0.8, it should be correct about 80% of the time for similar-confidence predictions. Calibration is important because the reward differences will be used as optimization signals. A well-calibrated model produces reliable gradients: large reward differences indicate strong preferences, while small differences reflect uncertainty. Poor calibration in either direction is problematic. Overconfidence means the model assigns large margins to pairs it only weakly prefers, which can cause the RL optimizer to over-exploit noisy reward signals. Underconfidence means the model assigns small margins even to clear quality differences, which can make the RL optimization slow or unstable.

Beyond automatic metrics, direct comparison with human evaluations provides insight into model quality. These evaluations are more expensive but can reveal failure modes that automatic metrics miss.

Correlation with human scores offers a continuous measure of alignment. If human evaluators rate responses on a Likert scale (1-5), we can measure Spearman correlation between model rewards and human ratings. Spearman correlation is appropriate here because we care about the ranking order rather than the absolute values of the scores. A correlation of 0.7 or higher with experienced human evaluators is generally considered good performance, though the threshold depends on the task difficulty and the consistency of the human ratings themselves.

Head-to-head win rates provide a complementary evaluation. Show humans new responses ranked by the reward model and ask them to validate the rankings. This catches cases where the model has learned spurious correlations. For example, if the model systematically overscores responses that use bullet points, head-to-head evaluation with human judges will reveal this: the judges will find that the model's highly-scored bulleted responses are often no better than the lower-scored prose responses.

It is also valuable to assess the reward model against the specific human annotators whose preferences it was trained on. If the model achieves high correlation with annotator A's preferences but low correlation with annotator B's, this reveals that the model has implicitly learned to weight different annotators' judgments differently. This can be a problem or a feature, depending on the relative quality of different annotators, but it should be understood rather than ignored.

Reward models can learn shortcuts that do not align with true quality. Identifying these shortcuts before RL training begins can prevent the policy from being steered in undesirable directions.

Length bias is one of the most common shortcuts. Models often prefer longer responses, even when brevity is more appropriate. This occurs because annotators, when uncertain about which response is better, often prefer the more detailed-seeming one, and response length correlates imperfectly with detail. Test by constructing pairs where the shorter response is clearly superior (e.g., a concise correct answer vs. a verbose but partially incorrect one) and checking whether the reward model correctly scores the concise response higher.

Style over substance is another pervasive issue. Models may prefer responses with confident tone, specific formatting (headers, numbered lists), or polished writing regardless of accuracy. Test with factually incorrect but confidently-written responses against accurate but plainly-written ones. If the reward model systematically prefers the confident-but-wrong response, style has contaminated the quality signal.

Prompt sensitivity reveals inconsistencies that could be exploited. Check if reward differences are consistent across paraphrased prompts asking the same question. If response A is preferred over response B when the prompt is "What is photosynthesis?" but B is preferred when the prompt is "Explain how plants make food," the reward model has learned prompt-specific patterns rather than a general quality function. This type of inconsistency is especially dangerous because the policy can learn to exploit it by generating responses that happen to work well with specific prompt phrasings.

Sycophancy is a particular concern for reward models trained on human feedback. Human annotators often prefer responses that agree with them or praise their question, even when such agreement is not warranted. A sycophantic reward model will score flattering or agreeable responses higher than honest but potentially unwelcome responses, which can steer the policy toward telling users what they want to hear rather than what is accurate.

These weaknesses become necessary during RL training, when the policy model can exploit them. We'll explore this issue in depth in the upcoming chapter on reward hacking.