Bradley-Terry Model: Converting Preferences to Rankings

In the previous chapter, we explored how human preference data is collected through pairwise comparisons: annotators see two model responses and indicate which one they prefer. But raw preference counts aren't directly useful for training. If response A was preferred over response B in 73% of comparisons, and response B was preferred over response C in 68% of comparisons, how much better is A than C? How do we convert these pairwise judgments into consistent quality scores that can guide model optimization? The answer requires a statistical model that can synthesize noisy, local comparisons into a coherent global ordering.

The Bradley-Terry model, developed by Ralph Bradley and Milton Terry in 1952, provides an elegant statistical framework for exactly this problem. Originally designed for ranking chess players and sports teams, it assumes each item has a latent "strength" parameter, and the probability of one item beating another depends only on the ratio of their strengths. This simple assumption leads to a powerful model that converts noisy pairwise comparisons into consistent, transitive rankings. Think of it as a hidden scoreboard: each competitor carries an invisible numeric quality score, and every observed comparison is a noisy measurement of the true gap between those scores.

The model's power comes from its probabilistic nature. Rather than asserting that the better item always wins, the Bradley-Terry model acknowledges that comparisons are inherently noisy. Two responses might both be reasonable answers to a question, and different annotators might prefer different ones depending on their background, values, or attention level on any given day. By modeling this uncertainty explicitly, the model can still recover a consistent underlying ranking even when individual comparisons contain apparent contradictions. The noise is not a nuisance to be eliminated; it is a basic part of human preference that the model embraces.

For LLM alignment, the Bradley-Terry model is foundational. As we'll see in the upcoming chapter on Reward Modeling, the reward model training objective is directly derived from the Bradley-Terry likelihood. Understanding this model deeply will clarify why reward models are trained the way they are, and why later techniques like Direct Preference Optimization (DPO) can eliminate the need for explicit reward modeling altogether. The Bradley-Terry model provides the statistical spine that connects raw human annotations to gradient updates in a neural network.

You might wonder why a model from 1952 would be relevant to modern language AI. The answer is that the basic problem it solves, turning pairwise comparisons into a global ranking, has not changed. What has changed is the scale and the nature of the items being compared. Instead of ranking eight chess players, we need to rank responses generated by language models across billions of possible prompts. The Bradley-Terry framework scales to this challenge because its parameters can be replaced by a neural network, preserving the mathematical structure while gaining the ability to generalize across inputs. This connection between a mid-century statistical model and 21st-century neural network training is one of the most elegant links in the history of machine learning.

Before diving into the mathematics, it helps to build strong intuition about what the model is doing at a high level. We start with a collection of items (model responses, chess players, restaurant dishes) and a set of pairwise comparisons where one item is declared the winner. Our goal is to assign each item a quality score such that these scores best explain the observed comparison outcomes. The Bradley-Terry model says: assume each item has a latent strength, and the probability of winning a comparison is a simple function of the two items' strengths. We then find the strength assignments that maximize the probability of having observed exactly the data we did. This is maximum likelihood estimation, and it produces the most statistically efficient ranking given our observations.

Consider a setting where we have $n$ items (in our case, model responses) that we want to rank based on human preferences. We don't observe quality scores directly; instead, we observe the outcomes of pairwise comparisons. When comparing items $i$ and $j$, a human annotator chooses one as better. This setup reflects how humans naturally make judgments: rather than assigning absolute quality scores on some numerical scale, people find it far easier to simply say which of two options they prefer. The challenge lies in synthesizing these individual binary decisions into a coherent global ranking. Think of this like reading a book of movie reviews where every review says only "I liked this film better than that one," and you must reconstruct a ranked list of every film ever made.

The core difficulty is that pairwise comparisons give us only local, relative information. Comparison A-vs-B tells us something about A relative to B, but nothing about where either item falls on an absolute scale. Comparison B-vs-C tells us something about B relative to C. If we want to know how A compares to C without having directly compared them, we need a model that can chain these local relationships into global conclusions. This is the inference problem that Bradley-Terry solves: it uses the entire structure of the comparison graph to find the parameter values that best explain all observed outcomes simultaneously.

Human preferences are also inherently noisy. The same two model responses shown to different annotators might produce different preference outcomes, because different annotators bring different expertise, different values, and different interpretations of the comparison criteria. Even the same annotator might decide differently on two separate occasions, especially for close pairs where neither response is clearly superior. A good preference model must account for this noise rather than pretending that comparisons are deterministic. The Bradley-Terry model handles noise by treating comparisons as Bernoulli trials, where the strength parameters determine the bias of the coin but cannot guarantee any particular outcome.

The key insight of the Bradley-Terry model is that we can explain these observed preferences through latent strength parameters. Each item $i$ has an associated strength $\pi_i > 0$ that represents its underlying quality. We never observe these strengths directly. Instead, we must infer them from the pattern of wins and losses across many comparisons. Items with higher strength values are better, and when two items are compared, the one with higher strength is more likely to win, but not guaranteed to win, which accounts for noise in human judgments. This probabilistic view is needed: it acknowledges that human preferences contain randomness, whether from inconsistency, variation in attention, or differences of opinion among annotators.

The basic assumption is that the probability of item $i$ being preferred over item $j$ depends only on the ratio of their strengths. We write this as:

where:

• $P(i \succ j)$: the probability that item $i$ is preferred over item $j$
• $\pi_i, \pi_j$: the latent strength parameters of items $i$ and $j$ (must be positive)

This formula has an intuitive interpretation that makes it easy to reason about. Consider first the case where $\pi_i = \pi_j$: substituting into the formula gives $P(i \succ j) = \frac{\pi_i}{2\pi_i} = 0.5$. This confirms our intuition that equally strong items have equal probability of being chosen. Now consider a more extreme case: if $\pi_i = 3\pi_j$, then $P(i \succ j) = \frac{3\pi_j}{3\pi_j + \pi_j} = \frac{3}{4} = 0.75$. An item three times as strong wins 75% of the time. Notice that the absolute magnitudes of the strengths do not matter, only their ratio. Whether we have strengths of 3 and 1, or 300 and 100, the predicted win probability remains the same. This ratio-based formulation captures the intuitive notion that what matters is relative quality, not absolute quality on some universal scale.

The formula also has a satisfying symmetry property: $P(i \succ j) + P(j \succ i) = \frac{\pi_i}{\pi_i + \pi_j} + \frac{\pi_j}{\pi_i + \pi_j} = 1$. This means the probabilities of the two outcomes of a comparison sum to one, as they must for a valid probability model. The stronger the item relative to its opponent, the closer its win probability gets to 1, but it can never reach 1 (which would require infinite relative strength). Even the weakest item has some probability of winning a comparison against the strongest. This reflects the reality that human judgments always have some element of unpredictability.

The curve in this figure reveals an important practical implication: the Bradley-Terry model becomes less informative about exact strength differences when items are very far apart in quality. The difference between a strength ratio of 8 and a ratio of 10 translates to almost no change in win probability, since both are close to 1. This saturation effect means that if you only compare very strong items against very weak ones, you learn little about the precise relative ordering of the strong items among themselves, or the weak items among themselves. Good experimental design for Bradley-Terry estimation should pair items of similar strength to get the most informative comparisons.

While the ratio formulation is intuitive, a more convenient parameterization uses log-strengths. This transformation opens up powerful connections to logistic regression and makes optimization considerably easier. It also reveals the deep mathematical structure that links Bradley-Terry to the neural network training objectives we will use for reward models. Let $\beta_i = \log(\pi_i)$. Since strengths must be positive, the log-strength can be any real number, which is mathematically more convenient for optimization algorithms that operate over unconstrained real-valued spaces. We can now derive the equivalent expression in terms of these log-strengths:

where:

• $\beta_i, \beta_j$: the log-strength parameters of items $i$ and $j$
• $\pi_i, \pi_j$: the latent strength parameters ($\beta = \log \pi$)
• $\sigma(x)$: the sigmoid function $\frac{1}{1 + e^{-x}}$

The function $\sigma(x)$ is the standard sigmoid familiar from logistic regression and neural network activations (as covered in our chapter on activation functions). The key insight here is that the log transformation converts a ratio of two quantities into a difference of their logs. The win probability depends not on the absolute strengths but on how much larger one log-strength is than the other. A difference of $\beta_i - \beta_j = 1$ means item $i$ is $e \approx 2.72$ times stronger than item $j$, and this corresponds to a win probability of $\sigma(1) \approx 0.73$. A difference of 2 means item $i$ is $e^2 \approx 7.39$ times stronger, corresponding to a win probability of $\sigma(2) \approx 0.88$.

The sigmoid function has several properties that make it ideal for representing probabilities. It maps any real number to the interval $(0, 1)$. This keeps valid probabilities. It is symmetric around zero, meaning $\sigma(-x) = 1 - \sigma(x)$, which captures the natural symmetry of pairwise comparisons: the probability that $j$ beats $i$ equals one minus the probability that $i$ beats $j$. The sigmoid is also smooth and differentiable everywhere, which enables gradient-based optimization. This differentiability is important when the parameters $\beta$ are outputs of a neural network: we can backpropagate gradients through the sigmoid to update the network weights.

Think of the log-strength parameterization as converting the strength problem into a distance problem. Each item is placed on a one-dimensional number line by its log-strength value. The probability that item $i$ beats item $j$ is a function of the signed distance $\beta_i - \beta_j$: positive means $i$ is to the right of $j$ on the line and thus more likely to win, negative means $j$ is more likely to win. The sigmoid converts this signed distance into a probability. The further apart two items are on this line, the more predictable their comparison outcomes become. Items clustered near the same point on the line will have comparison outcomes close to 50/50.

This log-linear form reveals that the Bradley-Terry model is equivalent to logistic regression where the probability of $i$ beating $j$ depends on the difference in their log-strengths. The sigmoid function ensures probabilities stay between 0 and 1, with the midpoint at 0.5 when $\beta_i = \beta_j$. This equivalence to logistic regression lets us use decades of research on efficient optimization algorithms, regularization techniques, and statistical theory about confidence intervals and hypothesis tests. Maximum likelihood estimation for Bradley-Terry is a well-studied convex optimization problem with guaranteed convergence to a global optimum.

A important property of the Bradley-Terry model is that it produces transitive preferences. Transitivity means that if A is better than B, and B is better than C, then A must be better than C. If we know how often A beats B, and how often B beats C, the model automatically determines how often A beats C. This follows directly from the parameterization, and understanding why reveals something deep about the model's structure. Think of transitivity as the model insisting that quality lives on a one-dimensional line: if A is to the right of B, and B is to the right of C, then A must be to the right of C. There is no room for circular orderings in a linear arrangement.

The transitivity arises because each item has a single strength parameter that governs all of its comparisons. When we estimate $\beta_A$, $\beta_B$, and $\beta_C$, these values must simultaneously explain the A-versus-B comparisons, the B-versus-C comparisons, and the A-versus-C comparisons. The model cannot assign different "strengths" to A depending on which item it is facing. This constraint forces global consistency. Every comparison provides information that constrains all the parameters jointly, and the maximum likelihood estimates balance all this information to find the single assignment of positions on the number line that best explains everything we observed.

This global consistency is what makes the model so powerful for synthesizing large amounts of comparison data. With $n$ items, there are $\binom{n}{2}$ possible pairwise matchups, but only $n-1$ free parameters (since one parameter is the anchor). The model over-constrains the problem: we use many more observations than parameters, and the maximum likelihood estimate finds the parameter values that best reconcile all these constraints simultaneously. This over-determination is a feature, not a bug. It means that comparisons between pairs we have observed well can inform predictions about pairs we have observed rarely, through the chain of transitivity.

Consider three items with strengths $\beta_A = 2$, $\beta_B = 1$, and $\beta_C = 0$. The pairwise probabilities are:

where:

• $\beta_A, \beta_B, \beta_C$: the example strengths ($2, 1, 0$)
• $P(A \succ B)$: the probability A beats B
• $\sigma(\cdot)$: the sigmoid function applied to strength differences

Notice that even though A and B have the same preference probability over their respective "next" item (both beat their weaker neighbor about 73% of the time), A has a much higher probability of beating C than B does of beating C. The strengths compose properly: the gap between A and C is the sum of the gaps between A and B and between B and C. Specifically, $\beta_A - \beta_C = (\beta_A - \beta_B) + (\beta_B - \beta_C) = 1 + 1 = 2$. This additive property of log-strengths is what makes the model so elegant. Strength differences accumulate predictably across chains of comparisons. The model implicitly answers the question "If A usually beats B and B usually beats C, how often should A beat C?" without needing us to collect A-versus-C comparisons directly.

This transitivity is a feature, not a bug. It forces the model to find a consistent ranking that best explains all observed comparisons, even when individual comparisons contain noise or contradictions. If humans sometimes prefer B over A, the model can still conclude A is better overall if A wins more often than it loses. The model essentially averages over the noise to find the underlying signal. When we observe an intransitivity in the data (say, A beats B 70% of the time, B beats C 70% of the time, but C beats A 55% of the time), the model will not try to represent this cycle explicitly. Instead, it will find the one-dimensional ordering that minimizes the total surprise across all observed outcomes. This averaging behavior is highly desirable for LLM evaluation: no single human comparison should be treated as ground truth, but the aggregate pattern across thousands of comparisons should be trusted.

However, as we will discuss in the Limitations section, this built-in transitivity can also be a constraint when human preferences violate the one-dimensional ordering assumption. Different annotators may prefer different attributes of responses, and these multi-dimensional preferences can produce systematic intransitivities that a Bradley-Terry model cannot capture. For now, it is enough to note that transitivity is both what makes the model tractable and what limits its expressiveness.

To estimate the strength parameters $\beta = (\beta_1, \beta_2, \ldots, \beta_n)$ from observed comparisons, we use maximum likelihood estimation. The goal is to find the parameter values that make the observed data most probable. This is a principled statistical approach: among all possible strength assignments, we seek the one that best explains the observed comparisons. Think of it as asking: "If I had to design a world where these strength values are true, which world would most plausibly produce the specific comparison outcomes I observed?"

Maximum likelihood estimation has excellent statistical properties for this problem. The estimates are consistent, meaning they converge to the true parameters as we collect more data. They are asymptotically efficient, meaning they extract the maximum possible information from the data. And the optimization problem is convex under mild conditions, meaning gradient-based methods will always find the global optimum rather than a local one. These properties make Bradley-Terry both a theoretically sound and computationally convenient choice for preference learning.

Suppose we have a dataset of $m$ pairwise comparisons, where comparison $k$ involves items $i_k$ and $j_k$, and item $w_k$ was chosen as the winner. The key assumption is that each comparison is independent, which is reasonable if different annotators make different comparisons, or if the same annotator's judgments don't depend on what they saw before. Under this independence assumption, the probability of the entire dataset is the product of the probabilities of each individual comparison outcome.

The likelihood of observing this data under the Bradley-Terry model is:

where:

• $L(\beta)$: the likelihood of the observed dataset given parameters $\beta$
• $m$: the total number of pairwise comparisons
• $w_k, \ell_k$: indices of the winning and losing items in the $k$-th comparison
• $\sigma(\cdot)$: the sigmoid function mapping the strength difference to a probability

The product form arises from independence: the probability of seeing all the comparisons we saw is the product of the probabilities of each individual comparison. Each term in the product is the Bradley-Terry probability that the actual winner wins, given the current strength parameters. The goal of maximum likelihood estimation is to find the $\beta$ values that make this product as large as possible. When the parameters correctly capture the true underlying strengths, the outcomes we observe will be the likely ones, making each factor close to 1 and the product large. When parameters are wrong, many observed outcomes will be surprises under the model, making the product small.

Taking the log-likelihood converts the product into a sum, which is more convenient both analytically and computationally. Logarithms are monotone transformations, so the $\beta$ that maximizes the log-likelihood is identical to the one that maximizes the likelihood itself:

where:

• $\log L(\beta)$: the log-likelihood function to be maximized
• $m$: the total number of comparisons
• $w_k, \ell_k$: indices of the winning and losing items in comparison $k$
• $\beta_{w_k}, \beta_{\ell_k}$: log-strength parameters for the winner and loser
• $-\log(1 + e^{-x})$: the log-sigmoid, equivalent to the negative softplus function

This is the negative of the binary cross-entropy loss where we predict that the winner should win. The connection to binary cross-entropy is important: it means that training a Bradley-Terry model is computationally identical to training a logistic regression classifier. The maximum likelihood estimates are found by maximizing this expression with respect to $\beta$, typically using gradient-based optimization methods like gradient descent or L-BFGS. The objective function is concave in $\beta$, which guarantees that gradient ascent will converge to the global maximum.

To understand the gradient intuitively, consider a single comparison where item $i$ beat item $j$, and suppose our current estimate assigns them equal strength ($\beta_i = \beta_j$). The model predicts a 50% win probability for item $i$, but we observed that item $i$ won. The gradient will push $\beta_i$ up and $\beta_j$ down, increasing the predicted probability that $i$ beats $j$ to better match the observation. The gradient magnitude is largest when the model is most surprised (predicted probability far from 1 for a win or far from 0 for a loss) and smallest when the model correctly assigns high probability to the observed outcome. This self-correcting behavior is what makes gradient-based optimization work so well here.

There's a subtlety that arises when we try to estimate Bradley-Terry parameters: the model has a location invariance. If we add a constant $c$ to all parameters, the probabilities don't change at all:

where:

• $c$: an arbitrary constant added to all log-strength parameters
• $\beta_i, \beta_j$: the log-strength parameters
• $\sigma(\cdot)$: the sigmoid function

This invariance occurs because the sigmoid only depends on differences between parameters, not on their absolute values. The constant $c$ cancels out in the subtraction. Mathematically, this means the parameters are not uniquely identifiable; there are infinitely many solutions that produce the same likelihood. If $(\beta_1, \beta_2, \beta_3) = (2, 1, 0)$ is a solution, then so is $(102, 101, 100)$, and so is $(-5, -6, -7)$. All these solutions make identical predictions. A gradient descent optimizer will wander along this flat direction forever without converging to a unique point.

To resolve this ambiguity, we typically fix one parameter (e.g., $\beta_1 = 0$) or add a constraint like $\sum_i \beta_i = 0$. This anchoring doesn't affect the relative rankings or prediction probabilities. It simply pins down the arbitrary location of the parameter scale so that optimization algorithms converge to a unique solution. The choice of anchor is arbitrary: we could anchor any item at any value, and the relative differences would remain the same. In practice, the conventional choice for LLM evaluation is to normalize so that the average rating across all models equals some baseline value, such as 1000 in the original Elo system or 0 in a centered parameterization.

The identifiability issue has a practical implication for how we interpret Bradley-Terry parameters: we should never interpret the absolute value of any single strength parameter, only the differences between parameters. A strength of $\beta = 3$ means nothing by itself. What matters is that this item has a log-strength 2 units higher than an item with $\beta = 1$, which translates to a win probability of $\sigma(2) \approx 0.88$ in head-to-head comparisons.

If the Bradley-Terry model sounds familiar, it should. The Elo rating system, used to rank chess players and now common in competitive gaming and language model evaluation, is a Bradley-Terry model in disguise. Understanding this connection illuminates both systems and explains why Elo ratings have become standard for comparing language models on public leaderboards.

The Elo system was designed for practical use in tournament settings where ratings need to be updated incrementally after each game rather than recomputed from scratch with every new observation. Rather than running a global optimization over all historical comparisons, Elo uses a simple online update rule: after each game, the winner gains points and the loser loses points, with the amount proportional to how surprising the outcome was given the current ratings. If the expected winner wins, little rating change occurs. If the underdog wins, a large rating transfer happens. This online update rule is an approximation to the maximum likelihood gradient, making Elo an efficient algorithm for the same underlying Bradley-Terry model.

Elo ratings are typically scaled so that a 400-point difference corresponds to a 10:1 expected score ratio. This scaling was chosen for historical and practical reasons: it produces ratings that are easy to interpret and compare. A rating difference of 400 points means the stronger player is expected to score about 10 points for every 1 point the weaker player scores over many games.

In standard Elo, the expected score of player $A$ against player $B$ is:

where:

• $E_A$: the expected score (win probability) of player $A$
• $R_A, R_B$: the Elo ratings of players $A$ and $B$
• $400$: the scaling factor where a difference of 400 points implies a 10:1 odds ratio

The use of base-10 exponentials rather than natural exponentials is purely conventional: Arpad Elo chose this scaling when he developed the system in the 1960s because it produced convenient numbers for chess ratings. The mathematical structure is identical to Bradley-Terry; only the scaling differs.

We can rewrite this in terms of the sigmoid function by converting the base-10 exponential to base-$e$:

where:

• $E_A$: the expected score (win probability) for player $A$
• $\ln 10 \approx 2.30$: the natural logarithm of 10
• $\sigma(\cdot)$: the standard sigmoid function

This derivation demonstrates that the Elo rating is just a scaled version of the Bradley-Terry log-strength parameter. Specifically, if $\beta$ is the Bradley-Terry parameter, then the corresponding Elo rating is $R = \frac{400}{\ln 10} \beta \approx 173.7 \beta$. The mathematical structure is identical; only the units differ. Every Elo system is a Bradley-Terry model, and every Bradley-Terry model can be expressed as an Elo system with a particular scaling convention.

The connection matters because Elo-style ratings are now used to evaluate LLMs. Platforms like LMSYS Chatbot Arena use Elo ratings computed from human preference comparisons to rank models. When you vote for which model response you prefer, these votes become pairwise comparisons that update the models' Elo ratings. The Bradley-Terry framework provides the statistical foundation for these rankings. This keeps the ratings are consistent and interpretable. A model with a 200-point Elo advantage over another is predicted to win head-to-head comparisons about 76% of the time: a concrete, actionable interpretation that pure benchmark scores on fixed datasets cannot provide.

The Elo system also demonstrates an important practical consideration: rating systems must handle the problem of new entrants. When a new language model joins the arena, we have no comparison data for it, so we initialize it at some default rating. As it accumulates comparisons against known models, its rating converges toward the true value. How quickly this convergence happens depends on the K-factor parameter in the Elo update rule, which controls the magnitude of rating changes per comparison. The Bradley-Terry maximum likelihood framework does not have this initialization problem: it estimates all parameters jointly from all available comparisons, so new models with few comparisons get wide uncertainty intervals rather than noisy point estimates.

Let's work through a concrete numerical example to solidify these concepts and see how the mathematics plays out with real numbers from start to finish. This will make the abstract framework concrete before we move to the code implementation. Suppose we have three model responses (A, B, C) and the following comparison outcomes from human annotators:

We have 30 total comparisons across three matchups. Looking at the raw data, A beats B 70% of the time, B beats C 80% of the time, and A beats C 90% of the time. These observed frequencies give a rough sense of the ranking (A is best, C is worst), but they are not the Bradley-Terry parameters. The parameters must be inferred by maximum likelihood, and they will differ from the simple observed frequencies because they must be globally consistent.

Step 1: Set the anchor. We set $\beta_C = 0$ as our anchor. This choice is arbitrary, but it resolves the identifiability issue and allows us to interpret the other parameters as "how much better than C" each item is. With this convention, $\beta_A$ and $\beta_B$ become the only free parameters.

Step 2: Write the log-likelihood. The log-likelihood function aggregates all our comparison data:

where:

• $\log L$: the log-likelihood function aggregating all comparisons
• $7, 3, \dots$: the observed counts (e.g., A beats B 7 times)
• $\sigma(\cdot)$: the sigmoid function mapping strength differences to probabilities
• $\beta_A, \beta_B$: the unknown log-strength parameters
• $0$: the fixed anchor value for item C ($\beta_C = 0$)

Each term in this sum corresponds to a set of comparisons. The first term, $7\log\sigma(\beta_A - \beta_B)$, contributes to the likelihood every time A beats B, which happened 7 times. The second term accounts for the 3 times B beat A. Notice how the symmetry of preferences emerges: when A beats B, we use $\sigma(\beta_A - \beta_B)$, and when B beats A, we use $\sigma(\beta_B - \beta_A) = 1 - \sigma(\beta_A - \beta_B)$.

Step 3: Take derivatives. To find the maximum, we take partial derivatives and set them to zero. We rely on the derivative property of the log-sigmoid function: $\frac{d}{dx} \log \sigma(x) = 1 - \sigma(x)$. This elegant result comes from the chain rule and the fact that $\sigma'(x) = \sigma(x)(1-\sigma(x))$, giving $\frac{d}{dx}\log\sigma(x) = \frac{\sigma'(x)}{\sigma(x)} = 1 - \sigma(x)$.

Applying the chain rule to the terms involving $\beta_A$:

1. When A wins (e.g., against B): The term is $\log \sigma(\beta_A - \beta_B)$. The derivative with respect to $\beta_A$ is $1 - \sigma(\beta_A - \beta_B)$.
2. When A loses (e.g., against B): The term is $\log \sigma(\beta_B - \beta_A)$. The derivative with respect to $\beta_A$ is $(1 - \sigma(\beta_B - \beta_A)) \cdot (-1)$, which simplifies to $-\sigma(\beta_A - \beta_B)$.

Substituting the counts into the gradient equation for $\beta_A$:

where:

• $\frac{\partial \log L}{\partial \beta_A}$: the gradient of the log-likelihood with respect to $\beta_A$
• $\beta_A, \beta_B$: the log-strength parameters
• $\sigma(\cdot)$: the sigmoid function
• $1 - \sigma(\cdot)$: the derivative of the $\log \sigma(\cdot)$ term for wins
• $-\sigma(\cdot)$: the derivative of the $\log \sigma(\cdot)$ term for losses

Using the property that $1 - \sigma(x) = \sigma(-x)$, we can write this more compactly as:

where:

• $\sigma(\beta_B - \beta_A)$: the probability that B beats A
• $\sigma(-\beta_A)$: the probability that C beats A (equivalent to $1 - \sigma(\beta_A)$)
• $\beta_A, \beta_B$: the log-strength parameters
• $\sigma(\cdot)$: the sigmoid function

There is a beautiful interpretation of this gradient. The first term, $7\sigma(\beta_B - \beta_A)$, is 7 times the probability that B beats A under the current model. The second term, $3\sigma(\beta_A - \beta_B)$, is 3 times the probability that A beats B. At the maximum likelihood solution, the gradient equals zero, which means: the expected number of times B should beat A under the model equals the observed number (3), and the expected number of times A should beat B equals the observed number (7). This is the method of moments interpretation of maximum likelihood for Bradley-Terry: at the optimum, observed win counts equal expected win counts for each item.

Step 4: Solve numerically. The gradient equations don't have a closed-form solution because the sigmoid function is nonlinear. Unlike linear regression, where we can solve for parameters directly, the nonlinearity of the sigmoid means we must use numerical methods. We run gradient ascent (or equivalently, minimize the negative log-likelihood) using an efficient optimizer like L-BFGS. The convexity of the objective guarantees we will find the global maximum.

The true maximum likelihood estimates for our three-item example work out to approximately $\beta_A \approx 2.05$, $\beta_B \approx 0.82$, and $\beta_C = 0$ (fixed). Let's verify these make sense. The gap $\beta_A - \beta_B \approx 1.23$ gives $\sigma(1.23) \approx 0.77$, close to the observed 70% but adjusted upward by the model's use of A-vs-C information. The gap $\beta_B - \beta_C = 0.82$ gives $\sigma(0.82) \approx 0.69$, close to the observed 80% but adjusted downward by the indirect information flowing through A. The Bradley-Terry model reconciles all three matchups simultaneously to find the best global explanation.

We'll implement Bradley-Terry estimation from scratch using gradient descent, then show how it's applied in practice. The implementation closely mirrors the mathematics we derived above, which should make the connection between theory and code transparent.

The ratings align with our expectations: Item A has the highest rating. This reflects its dominance in the pairwise comparisons. A difference of approximately 200 points between A and B suggests a strong win probability for A, corresponding to roughly 76% expected wins in head-to-head matchups.

Let's visualize how the Bradley-Terry probability varies with the strength difference, and where our estimated matchups fall on this curve:

In practice, preference data comes as individual comparison records rather than aggregated counts. Large-scale LLM evaluation pipelines might receive thousands of individual annotation records, each showing which of two responses a specific annotator preferred to a specific prompt. Let's implement a more realistic version that handles raw comparison logs and scales to many items:

The estimated parameters closely recover the true values. This shows that the Bradley-Terry model can reliably learn latent strengths from noisy pairwise comparisons. The errors are small relative to the true parameter gaps, meaning the model correctly recovers the ranking and approximate win probabilities even with just 500 comparisons across 5 items.

The sample size plot reveals an important practical consideration for LLM evaluation. At 50 comparisons, the mean absolute error in log-strength is above 0.3, which translates to meaningful uncertainty in predicted win probabilities. At 500 comparisons, the error drops below 0.1, giving much more reliable estimates. At 2000 comparisons, the model has high accuracy. For a real evaluation platform with dozens or hundreds of models, getting 200 or more comparisons per model pair requires substantial annotation effort, which is why platforms like Chatbot Arena aggregate crowd-sourced votes from many users over extended time periods.

The Bradley-Terry model is the foundation of reward model training in RLHF. Understanding this connection reveals why reward models are trained the way they are and illuminates the mathematical underpinnings of alignment techniques. This is perhaps the most important section of this chapter for practitioners building LLM alignment pipelines: the reward model training objective is not arbitrary; it is a direct instantiation of the Bradley-Terry likelihood.

Recall from the previous chapter that preference data consists of pairs $(y_w, y_l)$ where $y_w$ is the preferred response and $y_l$ is the dispreferred response to the same prompt $x$. A reward model $r_\theta(x, y)$ assigns a scalar score to each prompt-response pair. This score represents the model's estimate of how good the response is for that particular prompt. The key insight is that we can interpret this reward as a Bradley-Terry strength parameter. Rather than having one fixed strength per item, we have a neural network that maps any (prompt, response) pair to a strength. This generalization is what allows the model to transfer from training prompts to novel prompts encountered at inference time.

Concretely, the probability that response $y_w$ is preferred over $y_l$ should follow the Bradley-Terry formula:

where:

• $y_w, y_l$: the preferred (winning) and dispreferred (losing) responses
• $x$: the prompt or context
• $r_\theta(x, y)$: the reward model parameterized by weights $\theta$
• $\sigma(\cdot)$: the sigmoid function converting the reward difference into a probability

This formulation says that the probability of preferring one response over another depends on the difference in their reward scores, passed through a sigmoid. Responses with higher rewards are more likely to be preferred, but the relationship is probabilistic, letting for noise in human judgments. Training the reward model to maximize this probability is equivalent to Bradley-Terry maximum likelihood estimation, except now the strength parameters are neural network outputs rather than fixed scalars. Instead of learning one number per item, we learn a function that can assign rewards to any prompt-response pair.

The loss function for reward model training is:

where:

• $\mathcal{L}(\theta)$: the loss function to minimize (negative log-likelihood)
• $\mathcal{D}$: the dataset of preference pairs
• $\mathbb{E}$: the expectation over samples drawn from the dataset
• $\sigma(\cdot)$: the sigmoid function
• $r_\theta(x, y)$: the reward model output for input $x$ and response $y$
• $y_w, y_l$: the preferred and dispreferred responses

This is exactly the Bradley-Terry negative log-likelihood, with the expectation taken over the preference dataset. The connection is direct: minimizing this loss trains the neural network to produce reward scores that best explain the observed human preferences according to the Bradley-Terry model. We'll explore the full details of reward model architecture and training in the next chapter, but understanding Bradley-Terry makes the training objective transparent: we're fitting a neural network to predict comparison outcomes, using the same statistical framework that has been used to rank chess players since the 1950s.

The neural reward model generalizes the classical Bradley-Terry model in one important way: it can handle items that have never been directly compared. In the classical setting with $n$ fixed items, we could only make predictions about pairs that were directly represented in the parameter space. The neural reward model can evaluate any response to any prompt, because it learns a general function rather than a lookup table. This generalization is what makes reward models practical for training language models, where the "items" being compared are generated text sequences from an enormous combinatorial space.

The Bradley-Terry model makes assumptions that do not always hold in practice, and being clear-eyed about these limitations is needed for using the model responsibly in LLM alignment. Understanding where the model can break down helps practitioners design better data collection procedures, interpret results with appropriate caution, and choose extensions or alternatives when the standard model is insufficient.

The most significant limitation is the independence from irrelevant alternatives (IIA) assumption. The probability that A beats B doesn't depend on what other items exist in the comparison set. This means if you're comparing responses A and B, introducing a third response C shouldn't change how humans compare A to B. In practice, human preferences show strong context effects. A response that seems excellent when compared against a poor one might seem only adequate when compared against an outstanding one. Annotators calibrate their judgments to the comparison set they have seen, and their implicit sense of "what constitutes a good response" shifts based on the distribution of examples they encounter. The Bradley-Terry model ignores these calibration effects, which can cause the estimated strengths to depend on which comparisons were collected and in what order, rather than which reflects stable underlying qualities.

The model also assumes that preferences are governed by a single latent dimension: each item has one "strength" value that determines all its comparisons. Human preferences for text are often multidimensional. A response might be more helpful but less safe, or more creative but less accurate, or more concise but less complete. Different annotators may weight these dimensions differently based on their background and priorities. A single scalar reward compresses these tradeoffs into one number, potentially losing important information about the structure of preferences. When the true preference space is multidimensional, the Bradley-Terry model will give inconsistent parameter estimates across different subpopulations of annotators, because what looks like "strength" in one dimension may look like "weakness" in another. Research into multi-objective RLHF attempts to address this limitation by learning separate reward models for attributes such as helpfulness and safety, with factual accuracy scored independently as well.

Another challenge is intransitivity in human preferences. The Bradley-Terry model enforces transitivity by construction: if A is stronger than B and B is stronger than C, then A must be stronger than C. But humans sometimes exhibit cycles: preferring A over B, B over C, yet C over A. These cycles can arise from different annotators having different preferences, or from the same annotator weighing different factors in different comparisons. They can also emerge from strategic annotation behavior, where annotators reason about which responses are "better" according to an implicit rubric that may itself be inconsistent. The Bradley-Terry model smooths over these inconsistencies by finding the best one-dimensional ordering, which is usually desirable when intransitivities reflect noise. But when intransitivities reflect disagreement among annotator subgroups about what makes a response good, smoothing them away loses information about the diversity of human preferences.

The model also requires careful attention to data collection design. Items that are never directly or indirectly compared cannot be placed on the same ranking scale. If models A and B are only compared against models from different eras, and never against each other, their relative ordering cannot be estimated. In practice, LLM evaluation platforms address this by ensuring that all models have some comparisons against a common set of reference models, creating a connected comparison graph. The statistical precision of estimates also depends heavily on the graph structure: items that are frequently compared against many others have well-estimated parameters, while items that appear in few comparisons have high uncertainty.

A subtler issue is position bias in pairwise comparison collection. When annotators see two responses, their choice may be influenced by which response was shown first or on which side of the screen. Research on human annotation has found systematic biases where annotators prefer the first or the longer response regardless of actual quality. The Bradley-Terry model treats all comparisons as unbiased measurements of underlying quality, so position biases in the annotation process will corrupt the estimated parameters. Careful data collection protocols randomize presentation order, and some research has proposed extended models that include a nuisance parameter for position bias.

Finally, the Bradley-Terry model produces point estimates of strength parameters without automatic uncertainty quantification. Two items with widely different amounts of comparison data will both produce single scalar estimates, but the uncertainty in the estimate with few comparisons is much larger. Platforms that use Elo-style ratings often acknowledge this by noting when a model has a small number of comparisons, but the rating itself does not encode this uncertainty. Bayesian extensions of the Bradley-Terry model, such as placing priors over the strength parameters, can produce posterior distributions that naturally represent uncertainty, allowing proper statistical comparisons between items with different amounts of data.

Despite these limitations, the Bradley-Terry model has had a large impact on LLM alignment. Its simplicity makes it computationally tractable to train reward models on millions of preference comparisons. The resulting reward scores provide the training signal for PPO and other policy optimization algorithms. More recently, the mathematical connection between reward models and the Bradley-Terry likelihood enabled Direct Preference Optimization, which we'll cover later in this part. DPO derives an implicit reward model from the policy itself, eliminating the need for separate reward model training while still optimizing the same Bradley-Terry objective. This connection lets us derive the family of RLHF and DPO training procedures from the same preference model.

The Bradley-Terry model provides a principled framework for converting pairwise comparison data into consistent quality scores. Its core assumptions are:

• Each item has a latent strength parameter $\beta_i$
• The probability that item $i$ beats item $j$ is $\sigma(\beta_i - \beta_j)$
• Parameters are learned by maximum likelihood estimation

Key insights from this chapter:

• Log-linear form: The Bradley-Terry probability is a sigmoid of the strength difference, which makes it equivalent to logistic regression on pairwise comparisons. This connection to a well-studied model enables efficient optimization and rich statistical theory.
• Transitivity by design: The model automatically produces transitive rankings by placing all items on a one-dimensional scale. This is both a feature (consistency) and a limitation (it cannot represent multidimensional preference structure or systematic intransitivities).
• Identifiability: Only differences in strength matter, so one parameter must be anchored. Absolute values have no meaning; only relative differences are interpretable.
• Connection to Elo: The Elo rating system is a scaled Bradley-Terry model with an online update rule approximating maximum likelihood. The 400-point scaling factor converts natural log-strengths to a conventional rating scale.
• Foundation for reward modeling: The reward model training objective is exactly the Bradley-Terry likelihood with neural network-parameterized strengths. Minimizing the binary cross-entropy loss on preference pairs is mathematically identical to fitting a Bradley-Terry model where each item's strength is a neural network output.
• Sample efficiency matters: Accurate parameter estimation requires on the order of hundreds of comparisons per item pair, which motivates the large-scale crowd-sourced annotation efforts used to build alignment datasets.

In the next chapter on Reward Modeling, we'll see how this framework scales to neural networks that can generalize across prompts and responses. This provides the dense training signal needed for RLHF. We'll also examine how the reward model architecture is designed, how to avoid reward hacking, and how the trained reward model interacts with the policy optimization step.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about the Bradley-Terry model and its role in preference learning.