Preference Evaluation › The Case for Pairwise Comparisons

Absolute rating scales suffer from several systematic biases. Annotators interpret "good" differently depending on their expertise, cultural background, and the distribution of examples they have seen recently. A response that earns a 4 from a generous annotator might receive a 2 from a strict one, even if both agree on the relative ordering of responses. This annotator calibration drift makes it difficult to compare ratings collected at different times or by different people.

The problem becomes particularly acute when evaluation campaigns span multiple days or involve distributed annotation teams. An annotator who begins a session after reviewing particularly poor model outputs may calibrate their scale differently than one who begins after reviewing strong baselines. Even the same annotator may drift over time as they acclimate to model capabilities or fatigue sets in. These shifts create systematic noise that can obscure real model differences or create spurious apparent improvements.

Consider a concrete scenario: you are running a two-week evaluation to compare a new model against a baseline. In the first week, annotators review mostly weak baseline outputs and anchor their "5" rating at a relatively low quality threshold. In the second week, after seeing better model outputs, they recalibrate upward. The new model, evaluated in week two, receives lower ratings not because it is worse but because the scale has shifted. This temporal confounding is invisible in the raw rating data and can easily be mistaken for evidence that the new model underperforms.

Pairwise comparisons eliminate calibration issues by forcing a relative judgment. When an annotator selects Response A over Response B, they communicate real information regardless of whether they would rate both as "excellent" or both as "terrible." This invariance to additive bias makes comparisons the gold standard for subjective evaluation. The mathematical intuition is straightforward: if an annotator consistently applies a personal offset $c$ to all their ratings, this constant cancels out when computing differences between models. While absolute ratings suffer from $R_i = \text{true}_i + \epsilon_i + c$, pairwise comparisons yield $\Delta R = (\text{true}_A - \text{true}_B) + (\epsilon_A - \epsilon_B)$, where the calibration constant $c$ has vanished.

The advantages of pairwise comparisons extend beyond just eliminating the constant offset. They also reduce the cognitive load on annotators. Rather than mentally constructing a mapping from abstract quality to a numerical scale, the annotator simply identifies which of two concrete outputs better satisfies a given criterion. This simpler task produces more consistent judgments, higher inter-annotator agreement, and lower dropout rates in annotation campaigns. Research in social choice theory confirms that binary choices tend to produce more reliable and reproducible results than rating scales, particularly when the evaluated concepts are complex or multi-dimensional.

However, comparisons introduce their own challenges. They are more expensive than absolute ratings (requiring two outputs per judgment), create potential for position bias (preferring the first option shown), and raise statistical questions about transitivity. If Model A beats Model B, and Model B beats Model C, we expect Model A to beat Model C, but real human preferences often violate this expectation due to context-dependent strengths. For instance, Model A might excel at creative writing but struggle with factual accuracy, while Model B balances both moderately, and Model C prioritizes factuality above all. In such cases, A defeats B on creative tasks, B defeats C on general quality, yet C defeats A on factual queries, creating a preference cycle that confounds scalar rankings.

This problem of intransitivity is not a pathological edge case; it is a near-universal feature of LLM evaluation. The reason is that "quality" is not one-dimensional. A response can be more helpful but less harmless, more concise but less thorough, more confident but less accurate. When the prompt distribution covers multiple task types, different models will show different profiles of relative strengths, and the aggregate ranking will reflect an average over these profiles. How informative that average is depends entirely on whether the question you are asking is "which model performs best on average" or "which model is best for my specific use case."