Preference Evaluation › Statistical Significance

Determining that Model A has a higher Elo rating than Model B is insufficient; we need to know whether this difference is statistically significant or within the margin of error.

Intuition about statistical significance in Elo contexts differs from classical hypothesis testing in one important way: the test is not about whether a single comparison's result was due to chance, but about whether the aggregate pattern of many comparisons gives reliable evidence of a true rating difference. With enough comparisons, even tiny true differences become statistically distinguishable. With too few comparisons, even large true differences may not reach significance. Understanding the relationship between effect size, sample size, and power is needed for designing evaluation studies that can detect the differences you care about.

Elo ratings are estimates with associated uncertainty. The asymptotic variance of the Bradley-Terry parameter estimates can be derived from the Fisher information matrix. For model $i$, the variance is approximately:

$$\text{Var}(\hat{\beta}_i) \approx \left( \sum_{j \neq i} \frac{n_{ij} e^{\beta_i + \beta_j}}{(e^{\beta_i} + e^{\beta_j})^2} \right)^{-1}$$

where:

• $\text{Var}(\hat{\beta}_i)$: the asymptotic variance of the maximum likelihood estimate for model $i$'s strength parameter
• $\hat{\beta}_i$: the estimated strength parameter for model $i$
• $n_{ij}$: the number of comparisons between model $i$ and model $j$
• The summation $\sum_{j \neq i}$ aggregates over all opponents $j$ that model $i$ has faced
• The term $\frac{e^{\beta_i + \beta_j}}{(e^{\beta_i} + e^{\beta_j})^2}$ is the variance contribution from each matchup, peaking when models are evenly matched

The Fisher information measures the curvature of the log-likelihood function at its maximum; sharper curvature implies more information about the parameter and thus lower variance. Notice that the variance contribution from each opponent is maximized when $\beta_i \approx \beta_j$, confirming the intuition that matches between evenly matched competitors give the most information. Conversely, matches against much stronger or much weaker opponents contribute little to the precision of the estimate.

In practice, bootstrap resampling gives a more reliable variance estimate, especially when the Bradley-Terry assumptions are violated (e.g., due to intransitive preferences or context effects). The bootstrap makes no parametric assumptions about the distribution of preferences, instead using the empirical distribution of observed outcomes to estimate sampling variability.

To compute a 95% confidence interval for the rating difference between models A and B:

1. Bootstrap the comparison data 1000 times, resampling with replacement
2. Fit Bradley-Terry parameters for each bootstrap sample
3. Calculate the empirical 2.5th and 97.5th percentiles of $\beta_A - \beta_B$

If the interval excludes zero, the difference is significant at $p < 0.05$.

This non-parametric approach accounts for both sampling variability and potential model misspecification. By refitting the model on resampled data, we capture the uncertainty arising from having finite comparison samples. Wide confidence intervals indicate that additional comparisons are needed to distinguish the models, while narrow intervals suggest high confidence in the observed ordering.

The practical implications of confidence intervals often surprise practitioners accustomed to reporting point estimates. In evaluation campaigns with limited budgets, it is common to find that all models' confidence intervals overlap substantially, meaning that no pairwise differences are statistically significant even if the point estimates suggest a clear ranking. This is not a failure of the evaluation; it is honest communication of the uncertainty inherent in limited data. The appropriate response is either to collect more comparisons or to acknowledge that the models are effectively tied at the current sample size.

A rough rule of thumb for power planning: detecting a 100-point Elo difference (approximately 64% win rate for the stronger model) at 80% power with $\alpha = 0.05$ requires roughly 150-200 comparisons between the two models. Detecting a 50-point difference (approximately 57% win rate) requires 600-800 comparisons. These numbers scale with the number of models in the evaluation because each model must participate in multiple pairs. Planning your sample size before data collection is needed for avoiding underpowered studies.

In live evaluation systems, you want to detect when a new model is definitively better than the current champion without running infinite comparisons. Sequential probability ratio tests (SPRT) give stopping rules that maintain false positive rates while minimizing the number of required judgments.

The SPRT continues collecting comparisons until the likelihood ratio exceeds a threshold determined by desired $\alpha$ (false positive) and $\beta$ (false negative) rates:

$$\Lambda = \frac{P(\text{data} | H_1)}{P(\text{data} | H_0)} > \frac{1-\beta}{\alpha}$$

where:

• $\Lambda$: the likelihood ratio comparing the probability of observed data under alternative vs. null hypotheses
• $P(\text{data} | H_1)$: the probability of observing the comparison results assuming Model A is truly better (alternative hypothesis)
• $P(\text{data} | H_0)$: the probability of observing the comparison results assuming no difference between models (null hypothesis)
• $\alpha$: the Type I error rate (false positive probability)
• $\beta$: the Type II error rate (false negative probability)
• $\frac{1-\beta}{\alpha}$: the decision threshold determining when sufficient evidence has been collected

Unlike fixed-horizon tests, which require specifying the sample size in advance, SPRT adapts to the evidence. If Model A is sharply superior, the test terminates early with a decisive verdict. If the models are nearly identical, the test may continue longer to accumulate sufficient evidence to distinguish them. This approach can reduce evaluation costs by 30-50% compared to fixed-horizon testing while maintaining the same statistical power.

The SPRT framework requires specifying the alternative hypothesis in concrete terms: for example, "Model A has a win rate of at least $p_1$" rather than the vague claim that Model A is better than Model B. This concreteness is useful, because it forces you to explain the minimum effect size you care about before seeing the data. An evaluation designed to detect a 60% win rate will have different stopping rules than one designed to detect a 70% win rate, and the choice should reflect the practical significance threshold for your deployment decision.