Bradley-Terry Model › Connection to Reward Modeling

The Bradley-Terry model is the foundation of reward model training in RLHF. Understanding this connection reveals why reward models are trained the way they are and illuminates the mathematical underpinnings of alignment techniques. This is perhaps the most important section of this chapter for practitioners building LLM alignment pipelines: the reward model training objective is not arbitrary; it is a direct instantiation of the Bradley-Terry likelihood.

Recall from the previous chapter that preference data consists of pairs $(y_w, y_l)$ where $y_w$ is the preferred response and $y_l$ is the dispreferred response to the same prompt $x$. A reward model $r_\theta(x, y)$ assigns a scalar score to each prompt-response pair. This score represents the model's estimate of how good the response is for that particular prompt. The key insight is that we can interpret this reward as a Bradley-Terry strength parameter. Rather than having one fixed strength per item, we have a neural network that maps any (prompt, response) pair to a strength. This generalization is what allows the model to transfer from training prompts to novel prompts encountered at inference time.

Concretely, the probability that response $y_w$ is preferred over $y_l$ should follow the Bradley-Terry formula:

$$P(y_w \succ y_l \mid x) = \sigma(r_\theta(x, y_w) - r_\theta(x, y_l))$$

where:

• $y_w, y_l$: the preferred (winning) and dispreferred (losing) responses
• $x$: the prompt or context
• $r_\theta(x, y)$: the reward model parameterized by weights $\theta$
• $\sigma(\cdot)$: the sigmoid function converting the reward difference into a probability

This formulation says that the probability of preferring one response over another depends on the difference in their reward scores, passed through a sigmoid. Responses with higher rewards are more likely to be preferred, but the relationship is probabilistic, letting for noise in human judgments. Training the reward model to maximize this probability is equivalent to Bradley-Terry maximum likelihood estimation, except now the strength parameters are neural network outputs rather than fixed scalars. Instead of learning one number per item, we learn a function that can assign rewards to any prompt-response pair.

The loss function for reward model training is:

$$\mathcal{L}(\theta) = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \left[ \log \sigma(r_\theta(x, y_w) - r_\theta(x, y_l)) \right]$$

where:

• $\mathcal{L}(\theta)$: the loss function to minimize (negative log-likelihood)
• $\mathcal{D}$: the dataset of preference pairs
• $\mathbb{E}$: the expectation over samples drawn from the dataset
• $\sigma(\cdot)$: the sigmoid function
• $r_\theta(x, y)$: the reward model output for input $x$ and response $y$
• $y_w, y_l$: the preferred and dispreferred responses

This is exactly the Bradley-Terry negative log-likelihood, with the expectation taken over the preference dataset. The connection is direct: minimizing this loss trains the neural network to produce reward scores that best explain the observed human preferences according to the Bradley-Terry model. We'll explore the full details of reward model architecture and training in the next chapter, but understanding Bradley-Terry makes the training objective transparent: we're fitting a neural network to predict comparison outcomes, using the same statistical framework that has been used to rank chess players since the 1950s.

The neural reward model generalizes the classical Bradley-Terry model in one important way: it can handle items that have never been directly compared. In the classical setting with $n$ fixed items, we could only make predictions about pairs that were directly represented in the parameter space. The neural reward model can evaluate any response to any prompt, because it learns a general function rather than a lookup table. This generalization is what makes reward models practical for training language models, where the "items" being compared are generated text sequences from an enormous combinatorial space.