Bradley-Terry Model › The Pairwise Comparison Framework

Consider a setting where we have $n$ items (in our case, model responses) that we want to rank based on human preferences. We don't observe quality scores directly; instead, we observe the outcomes of pairwise comparisons. When comparing items $i$ and $j$, a human annotator chooses one as better. This setup reflects how humans naturally make judgments: rather than assigning absolute quality scores on some numerical scale, people find it far easier to simply say which of two options they prefer. The challenge lies in synthesizing these individual binary decisions into a coherent global ranking. Think of this like reading a book of movie reviews where every review says only "I liked this film better than that one," and you must reconstruct a ranked list of every film ever made.

The core difficulty is that pairwise comparisons give us only local, relative information. Comparison A-vs-B tells us something about A relative to B, but nothing about where either item falls on an absolute scale. Comparison B-vs-C tells us something about B relative to C. If we want to know how A compares to C without having directly compared them, we need a model that can chain these local relationships into global conclusions. This is the inference problem that Bradley-Terry solves: it uses the entire structure of the comparison graph to find the parameter values that best explain all observed outcomes simultaneously.

Human preferences are also inherently noisy. The same two model responses shown to different annotators might produce different preference outcomes, because different annotators bring different expertise, different values, and different interpretations of the comparison criteria. Even the same annotator might decide differently on two separate occasions, especially for close pairs where neither response is clearly superior. A good preference model must account for this noise rather than pretending that comparisons are deterministic. The Bradley-Terry model handles noise by treating comparisons as Bernoulli trials, where the strength parameters determine the bias of the coin but cannot guarantee any particular outcome.

The key insight of the Bradley-Terry model is that we can explain these observed preferences through latent strength parameters. Each item $i$ has an associated strength $\pi_i > 0$ that represents its underlying quality. We never observe these strengths directly. Instead, we must infer them from the pattern of wins and losses across many comparisons. Items with higher strength values are better, and when two items are compared, the one with higher strength is more likely to win, but not guaranteed to win, which accounts for noise in human judgments. This probabilistic view is needed: it acknowledges that human preferences contain randomness, whether from inconsistency, variation in attention, or differences of opinion among annotators.

The basic assumption is that the probability of item $i$ being preferred over item $j$ depends only on the ratio of their strengths. We write this as:

where:

• $P(i \succ j)$: the probability that item $i$ is preferred over item $j$
• $\pi_i, \pi_j$: the latent strength parameters of items $i$ and $j$ (must be positive)

This formula has an intuitive interpretation that makes it easy to reason about. Consider first the case where $\pi_i = \pi_j$: substituting into the formula gives $P(i \succ j) = \frac{\pi_i}{2\pi_i} = 0.5$. This confirms our intuition that equally strong items have equal probability of being chosen. Now consider a more extreme case: if $\pi_i = 3\pi_j$, then $P(i \succ j) = \frac{3\pi_j}{3\pi_j + \pi_j} = \frac{3}{4} = 0.75$. An item three times as strong wins 75% of the time. Notice that the absolute magnitudes of the strengths do not matter, only their ratio. Whether we have strengths of 3 and 1, or 300 and 100, the predicted win probability remains the same. This ratio-based formulation captures the intuitive notion that what matters is relative quality, not absolute quality on some universal scale.

The formula also has a satisfying symmetry property: $P(i \succ j) + P(j \succ i) = \frac{\pi_i}{\pi_i + \pi_j} + \frac{\pi_j}{\pi_i + \pi_j} = 1$. This means the probabilities of the two outcomes of a comparison sum to one, as they must for a valid probability model. The stronger the item relative to its opponent, the closer its win probability gets to 1, but it can never reach 1 (which would require infinite relative strength). Even the weakest item has some probability of winning a comparison against the strongest. This reflects the reality that human judgments always have some element of unpredictability.

The curve in this figure reveals an important practical implication: the Bradley-Terry model becomes less informative about exact strength differences when items are very far apart in quality. The difference between a strength ratio of 8 and a ratio of 10 translates to almost no change in win probability, since both are close to 1. This saturation effect means that if you only compare very strong items against very weak ones, you learn little about the precise relative ordering of the strong items among themselves, or the weak items among themselves. Good experimental design for Bradley-Terry estimation should pair items of similar strength to get the most informative comparisons.