Preference Evaluation › The Elo Rating System

The Elo rating system, developed by Arpad Elo for chess rankings in the 1960s, gives an elegant solution for aggregating pairwise comparisons into a global skill rating. Each model maintains a rating $R$, and the difference in ratings predicts the expected outcome of a match. The system works because it converts the sparse, local information of individual match results into a global, transitive ranking that approximates the underlying skill distribution.

Elo's key insight was that skill differences should predict win probabilities, and that those predictions should be updated in proportion to how surprised we are by the actual outcome. If a model rated 1800 beats one rated 1200, we learn almost nothing new because that outcome was nearly certain. But if the 1200-rated model wins, we sharply revise our estimates of both models' true strengths. This asymmetric updating makes Elo a form of online Bayesian inference, continually refining our beliefs about the true ordering of competitors as new evidence arrives.

The Elo system has been adapted beyond chess to many competitive domains: video games (where TrueSkill and Glicko extend Elo's framework), academic competitions, and more recently to the evaluation of machine learning systems. The natural alignment between Elo's assumptions (players have latent skill levels that determine match outcomes probabilistically) and the structure of LLM evaluation (models have latent quality levels that determine preference outcomes) makes Elo a natural choice for preference leaderboards.

Given two models with ratings $R_A$ and $R_B$, the expected score for Model A is:

$$E_A = \frac{1}{1 + 10^{(R_B - R_A)/400}}$$

where:

• $E_A$: the expected score for Model A (probability of winning plus half the probability of drawing)
• $R_A$: the current Elo rating of Model A
• $R_B$: the current Elo rating of Model B
• $400$: the scale factor determining the rating-to-probability mapping (a 200-point difference yields approximately 76% expected win rate)

The expected score $E_A$ is the probability that Model A wins plus half the probability of a draw. The divisor of 400 creates a convenient scale where differences of 200 points correspond to roughly 3:1 odds, and differences of 400 points correspond to 10:1 odds. This logistic curve ensures that expected scores remain bounded between 0 and 1 while mapping linear rating differences to nonlinear probability changes. The specific value of 400 is conventional, chosen to match the historical chess rating scale, but the system works with any scale factor that matches the observed variance in your comparison data.

The formula is equivalent to a logistic function applied to the rating difference. As the difference $R_A - R_B$ grows, $E_A$ approaches 1 (near-certain win). As it shrinks toward zero, $E_A$ approaches 0.5 (coin flip). This sigmoid shape captures the intuition that skill differences become increasingly decisive as they grow larger, but that even large advantages do not guarantee victory against random variation in individual comparisons.

After observing an actual outcome $S_A$ (1 for win, 0.5 for tie, 0 for loss), we update the rating:

$$R_A' = R_A + K \cdot (S_A - E_A)$$

where:

• $R_A'$: the updated rating for Model A after the match
• $R_A$: the current rating of Model A before the match
• $K$: the K-factor, controlling the sensitivity of rating updates (higher values mean faster adaptation)
• $S_A$: the actual score observed for Model A (1 for win, 0.5 for tie, 0 for loss)
• $E_A$: the expected score for Model A calculated before the match

A large $K$ allows ratings to respond quickly to new information but introduces volatility. A small $K$ stabilizes ratings but slows adaptation to real improvements. The update rule has an intuitive interpretation: you adjust the rating based on the prediction error $(S_A - E_A)$. If the model performs better than expected, the rating increases; if worse, it decreases. The magnitude of change scales with how surprising the result was. A 1200-rated model defeating a 1500-rated model produces a larger update than a 1480-rated model defeating a 1500-rated model.

The K-factor update can be understood as gradient descent on the Bradley-Terry log-likelihood. The gradient of the log-likelihood with respect to the rating of Model A, given a comparison outcome, is proportional to $(S_A - E_A)$. The K-factor plays the role of the learning rate, controlling the step size of each update. This gradient descent interpretation reveals why large K-factors can destabilize ratings: just as an excessively large learning rate causes gradient descent to overshoot, a large K-factor causes ratings to oscillate around the true values rather than converging smoothly.

For LLM evaluation, typical K-factors range from 16 to 32. Online platforms like Chatbot Arena use dynamic K-factors that decrease as models accumulate more matches, mimicking Bayesian updating where we become more confident in established ratings. Early in a model's evaluation history, when uncertainty is high, large K-factors allow rapid convergence to the appropriate rating level. Once a model has participated in hundreds of matches, the K-factor decreases, preventing random fluctuations from perturbing an otherwise stable estimate.

The Elo update rule approximates maximum likelihood estimation for the Bradley-Terry model we discussed in Bradley-Terry Model. In the Bradley-Terry framework, the probability that model $i$ beats model $j$ is:

$$\begin{aligned} P(i \text{ beats } j) &= \frac{e^{\beta_i}}{e^{\beta_i} + e^{\beta_j}} \\ &= \frac{e^{\beta_i}/e^{\beta_i}}{(e^{\beta_i} + e^{\beta_j})/e^{\beta_i}} && \text{(divide numerator and denominator by } e^{\beta_i} \text{)} \\ &= \frac{1}{1 + e^{\beta_j}/e^{\beta_i}} && \text{(simplify)} \\ &= \frac{1}{1 + e^{-(\beta_i - \beta_j)}} && \text{(exponent rule: } e^{\beta_j}/e^{\beta_i} = e^{-(\beta_i - \beta_j)} \text{)} \end{aligned}$$

where:

• $P(i \text{ beats } j)$: the probability that model $i$ defeats model $j$ in a pairwise comparison
• $\beta_i$: the latent strength parameter for model $i$ (higher values indicate stronger models)
• $\beta_j$: the latent strength parameter for model $j$
• $e^{\beta_i}$: the exponential of the strength parameter, so positive values and amplifying differences

If we set $R_i = 400 \cdot \beta_i / \ln(10)$, this becomes exactly the Elo expected score formula. The Elo update is in effect an online stochastic gradient descent step on the Bradley-Terry log-likelihood, where $K$ is the learning rate. Each update moves the rating in the direction that would increase the likelihood of the observed outcome, with the step size controlled by $K$.

This connection means that Elo ratings, when stabilized, estimate the underlying preference parameters that would be obtained by fitting a Bradley-Terry model to the entire comparison matrix. The online updates give a computationally efficient approximation to the batch optimization problem, making Elo ideal for live leaderboard systems where new comparisons arrive continuously and ratings must update in real time.

The Bradley-Terry model rests on a core assumption: each model has a single latent strength that governs all comparisons, regardless of the specific prompt or task. This assumption is clearly an idealization. In practice, GPT-4 might outperform Claude on mathematical reasoning while Claude outperforms GPT-4 on creative writing. The Bradley-Terry model averages across these task-specific advantages and disadvantages, creating a scalar that captures performance over the empirical distribution of prompts in the evaluation set. Whether this scalar is real depends entirely on whether that prompt distribution matches the distribution you care about in deployment.