Bradley-Terry Model › Bradley-Terry Model: Converting Preferences to Rankings

In the previous chapter, we explored how human preference data is collected through pairwise comparisons: annotators see two model responses and indicate which one they prefer. But raw preference counts aren't directly useful for training. If response A was preferred over response B in 73% of comparisons, and response B was preferred over response C in 68% of comparisons, how much better is A than C? How do we convert these pairwise judgments into consistent quality scores that can guide model optimization? The answer requires a statistical model that can synthesize noisy, local comparisons into a coherent global ordering.

The Bradley-Terry model, developed by Ralph Bradley and Milton Terry in 1952, provides an elegant statistical framework for exactly this problem. Originally designed for ranking chess players and sports teams, it assumes each item has a latent "strength" parameter, and the probability of one item beating another depends only on the ratio of their strengths. This simple assumption leads to a powerful model that converts noisy pairwise comparisons into consistent, transitive rankings. Think of it as a hidden scoreboard: each competitor carries an invisible numeric quality score, and every observed comparison is a noisy measurement of the true gap between those scores.

The model's power comes from its probabilistic nature. Rather than asserting that the better item always wins, the Bradley-Terry model acknowledges that comparisons are inherently noisy. Two responses might both be reasonable answers to a question, and different annotators might prefer different ones depending on their background, values, or attention level on any given day. By modeling this uncertainty explicitly, the model can still recover a consistent underlying ranking even when individual comparisons contain apparent contradictions. The noise is not a nuisance to be eliminated; it is a basic part of human preference that the model embraces.

For LLM alignment, the Bradley-Terry model is foundational. As we'll see in the upcoming chapter on Reward Modeling, the reward model training objective is directly derived from the Bradley-Terry likelihood. Understanding this model deeply will clarify why reward models are trained the way they are, and why later techniques like Direct Preference Optimization (DPO) can eliminate the need for explicit reward modeling altogether. The Bradley-Terry model provides the statistical spine that connects raw human annotations to gradient updates in a neural network.

You might wonder why a model from 1952 would be relevant to modern language AI. The answer is that the basic problem it solves, turning pairwise comparisons into a global ranking, has not changed. What has changed is the scale and the nature of the items being compared. Instead of ranking eight chess players, we need to rank responses generated by language models across billions of possible prompts. The Bradley-Terry framework scales to this challenge because its parameters can be replaced by a neural network, preserving the mathematical structure while gaining the ability to generalize across inputs. This connection between a mid-century statistical model and 21st-century neural network training is one of the most elegant links in the history of machine learning.

Before diving into the mathematics, it helps to build strong intuition about what the model is doing at a high level. We start with a collection of items (model responses, chess players, restaurant dishes) and a set of pairwise comparisons where one item is declared the winner. Our goal is to assign each item a quality score such that these scores best explain the observed comparison outcomes. The Bradley-Terry model says: assume each item has a latent strength, and the probability of winning a comparison is a simple function of the two items' strengths. We then find the strength assignments that maximize the probability of having observed exactly the data we did. This is maximum likelihood estimation, and it produces the most statistically efficient ranking given our observations.