Preference Evaluation › Worked Example: Comparing Three Chatbots

Let us walk through a concrete example. Suppose we are evaluating three models: Alpha, Beta, and Gamma. We collect the following comparison results over 300 total matchups:

• Alpha vs Beta: Alpha wins 45, Beta wins 30, 5 ties
• Alpha vs Gamma: Alpha wins 35, Gamma wins 50, 5 ties
• Beta vs Gamma: Beta wins 40, Gamma wins 35, 5 ties

First, we initialize all ratings at 1500. Using $K=32$:

For Alpha vs Beta:

$$\begin{aligned} E_{\text{Alpha}} &= \frac{1}{1 + 10^{(1500-1500)/400}} && \text{(expected score with equal ratings)} \\ &= 0.5 \\[10pt] S_{\text{Alpha}} &= \frac{45 + 0.5 \cdot 5}{80} && \text{(actual score: wins + half ties)} \\ &= 0.594 \\[10pt] R_{\text{Alpha}}' &= 1500 + 32 \cdot (0.594 - 0.5) && \text{(rating update)} \\ &= 1503.0 \\[10pt] R_{\text{Beta}}' &= 1500 + 32 \cdot (0.406 - 0.5) \\ &= 1497.0 \end{aligned}$$

Continuing this process iteratively until convergence yields stable ratings. The model with the strongest performance (Alpha against Beta, but weak against Gamma) creates a cyclic preference: Alpha > Beta > Gamma > Alpha, known as intransitivity. Real LLM evaluations often exhibit such cycles when models have different strengths (e.g., one is more helpful but less harmless).

This cyclic preference highlights a basic limitation of scalar rankings. In our example, Alpha might excel at creative writing but struggle with factual accuracy, while Beta balances both moderately, and Gamma prioritizes factual precision above all else. Depending on the prompt distribution, any of these models could appear superior. Intransitivity suggests that the very notion of a single "best" model may be illusory when quality is multi-dimensional, and that Elo ratings should be interpreted as capturing average performance over a specific task distribution rather than universal superiority.

How should we interpret this result in practice? Rather than concluding that the ranking is meaningless, we should recognize it as useful within its scope: the rank captures average performance over the prompt set used in the evaluation. If that prompt set is the distribution of prompts our users will send, then the ranking is maximally relevant. If the prompt set is unrepresentative, the ranking is unrepresentative too. The solution is not to abandon the ranking but to ensure that the evaluation prompt distribution is carefully designed to match the deployment distribution.

The intransitivity also motivates stratified reporting. In addition to a global Elo ranking, reporting per-domain win rates (creative writing, factual Q&A, code generation, summarization) gives a richer picture of each model's strengths and helps users select the right model for their specific application. A model with a lower overall Elo might be the clear choice for a specific domain where it has a large win-rate advantage.