Bradley-Terry Model › Summary

The Bradley-Terry model provides a principled framework for converting pairwise comparison data into consistent quality scores. Its core assumptions are:

• Each item has a latent strength parameter $\beta_i$
• The probability that item $i$ beats item $j$ is $\sigma(\beta_i - \beta_j)$
• Parameters are learned by maximum likelihood estimation

Key insights from this chapter:

• Log-linear form: The Bradley-Terry probability is a sigmoid of the strength difference, which makes it equivalent to logistic regression on pairwise comparisons. This connection to a well-studied model enables efficient optimization and rich statistical theory.
• Transitivity by design: The model automatically produces transitive rankings by placing all items on a one-dimensional scale. This is both a feature (consistency) and a limitation (it cannot represent multidimensional preference structure or systematic intransitivities).
• Identifiability: Only differences in strength matter, so one parameter must be anchored. Absolute values have no meaning; only relative differences are interpretable.
• Connection to Elo: The Elo rating system is a scaled Bradley-Terry model with an online update rule approximating maximum likelihood. The 400-point scaling factor converts natural log-strengths to a conventional rating scale.
• Foundation for reward modeling: The reward model training objective is exactly the Bradley-Terry likelihood with neural network-parameterized strengths. Minimizing the binary cross-entropy loss on preference pairs is mathematically identical to fitting a Bradley-Terry model where each item's strength is a neural network output.
• Sample efficiency matters: Accurate parameter estimation requires on the order of hundreds of comparisons per item pair, which motivates the large-scale crowd-sourced annotation efforts used to build alignment datasets.

In the next chapter on Reward Modeling, we'll see how this framework scales to neural networks that can generalize across prompts and responses. This provides the dense training signal needed for RLHF. We'll also examine how the reward model architecture is designed, how to avoid reward hacking, and how the trained reward model interacts with the policy optimization step.