Bradley-Terry Model › The Bradley-Terry Likelihood

To estimate the strength parameters $\beta = (\beta_1, \beta_2, \ldots, \beta_n)$ from observed comparisons, we use maximum likelihood estimation. The goal is to find the parameter values that make the observed data most probable. This is a principled statistical approach: among all possible strength assignments, we seek the one that best explains the observed comparisons. Think of it as asking: "If I had to design a world where these strength values are true, which world would most plausibly produce the specific comparison outcomes I observed?"

Maximum likelihood estimation has excellent statistical properties for this problem. The estimates are consistent, meaning they converge to the true parameters as we collect more data. They are asymptotically efficient, meaning they extract the maximum possible information from the data. And the optimization problem is convex under mild conditions, meaning gradient-based methods will always find the global optimum rather than a local one. These properties make Bradley-Terry both a theoretically sound and computationally convenient choice for preference learning.

Suppose we have a dataset of $m$ pairwise comparisons, where comparison $k$ involves items $i_k$ and $j_k$, and item $w_k$ was chosen as the winner. The key assumption is that each comparison is independent, which is reasonable if different annotators make different comparisons, or if the same annotator's judgments don't depend on what they saw before. Under this independence assumption, the probability of the entire dataset is the product of the probabilities of each individual comparison outcome.

The likelihood of observing this data under the Bradley-Terry model is:

$$L(\beta) = \prod_{k=1}^{m} P(w_k \text{ wins comparison } k) = \prod_{k=1}^{m} \sigma(\beta_{w_k} - \beta_{\ell_k})$$

where:

• $L(\beta)$: the likelihood of the observed dataset given parameters $\beta$
• $m$: the total number of pairwise comparisons
• $w_k, \ell_k$: indices of the winning and losing items in the $k$-th comparison
• $\sigma(\cdot)$: the sigmoid function mapping the strength difference to a probability

The product form arises from independence: the probability of seeing all the comparisons we saw is the product of the probabilities of each individual comparison. Each term in the product is the Bradley-Terry probability that the actual winner wins, given the current strength parameters. The goal of maximum likelihood estimation is to find the $\beta$ values that make this product as large as possible. When the parameters correctly capture the true underlying strengths, the outcomes we observe will be the likely ones, making each factor close to 1 and the product large. When parameters are wrong, many observed outcomes will be surprises under the model, making the product small.

Taking the log-likelihood converts the product into a sum, which is more convenient both analytically and computationally. Logarithms are monotone transformations, so the $\beta$ that maximizes the log-likelihood is identical to the one that maximizes the likelihood itself:

$$\begin{aligned} \log L(\beta) &= \sum_{k=1}^{m} \log \sigma(\beta_{w_k} - \beta_{\ell_k}) && \text{(log of product is sum of logs)} \\ &= \sum_{k=1}^{m} \log \left( \frac{1}{1 + e^{-(\beta_{w_k} - \beta_{\ell_k})}} \right) && \text{(substitute sigmoid definition)} \\ &= -\sum_{k=1}^{m} \log(1 + e^{-(\beta_{w_k} - \beta_{\ell_k})}) && \text{(using } \log(1/x) = -\log x \text{)} \end{aligned}$$

where:

• $\log L(\beta)$: the log-likelihood function to be maximized
• $m$: the total number of comparisons
• $w_k, \ell_k$: indices of the winning and losing items in comparison $k$
• $\beta_{w_k}, \beta_{\ell_k}$: log-strength parameters for the winner and loser
• $-\log(1 + e^{-x})$: the log-sigmoid, equivalent to the negative softplus function

This is the negative of the binary cross-entropy loss where we predict that the winner should win. The connection to binary cross-entropy is important: it means that training a Bradley-Terry model is computationally identical to training a logistic regression classifier. The maximum likelihood estimates are found by maximizing this expression with respect to $\beta$, typically using gradient-based optimization methods like gradient descent or L-BFGS. The objective function is concave in $\beta$, which guarantees that gradient ascent will converge to the global maximum.

To understand the gradient intuitively, consider a single comparison where item $i$ beat item $j$, and suppose our current estimate assigns them equal strength ($\beta_i = \beta_j$). The model predicts a 50% win probability for item $i$, but we observed that item $i$ won. The gradient will push $\beta_i$ up and $\beta_j$ down, increasing the predicted probability that $i$ beats $j$ to better match the observation. The gradient magnitude is largest when the model is most surprised (predicted probability far from 1 for a win or far from 0 for a loss) and smallest when the model correctly assigns high probability to the observed outcome. This self-correcting behavior is what makes gradient-based optimization work so well here.

There's a subtlety that arises when we try to estimate Bradley-Terry parameters: the model has a location invariance. If we add a constant $c$ to all parameters, the probabilities don't change at all:

$$\sigma((\beta_i + c) - (\beta_j + c)) = \sigma(\beta_i - \beta_j)$$

where:

• $c$: an arbitrary constant added to all log-strength parameters
• $\beta_i, \beta_j$: the log-strength parameters
• $\sigma(\cdot)$: the sigmoid function

This invariance occurs because the sigmoid only depends on differences between parameters, not on their absolute values. The constant $c$ cancels out in the subtraction. Mathematically, this means the parameters are not uniquely identifiable; there are infinitely many solutions that produce the same likelihood. If $(\beta_1, \beta_2, \beta_3) = (2, 1, 0)$ is a solution, then so is $(102, 101, 100)$, and so is $(-5, -6, -7)$. All these solutions make identical predictions. A gradient descent optimizer will wander along this flat direction forever without converging to a unique point.

To resolve this ambiguity, we typically fix one parameter (e.g., $\beta_1 = 0$) or add a constraint like $\sum_i \beta_i = 0$. This anchoring doesn't affect the relative rankings or prediction probabilities. It simply pins down the arbitrary location of the parameter scale so that optimization algorithms converge to a unique solution. The choice of anchor is arbitrary: we could anchor any item at any value, and the relative differences would remain the same. In practice, the conventional choice for LLM evaluation is to normalize so that the average rating across all models equals some baseline value, such as 1000 in the original Elo system or 0 in a centered parameterization.

The identifiability issue has a practical implication for how we interpret Bradley-Terry parameters: we should never interpret the absolute value of any single strength parameter, only the differences between parameters. A strength of $\beta = 3$ means nothing by itself. What matters is that this item has a log-strength 2 units higher than an item with $\beta = 1$, which translates to a win probability of $\sigma(2) \approx 0.88$ in head-to-head comparisons.