Preference Evaluation › A/B Comparison Design

Designing a preference evaluation requires careful attention to experimental protocol. The simplest form presents two anonymized outputs side-by-side and asks the annotator to choose the better one, with optional buttons for "tie" or "uncertain." But even this simple design embeds dozens of choices that affect the validity and statistical power of your results. The best evaluation pipelines treat comparison design as a scientific experiment, applying the same rigor to randomization, blinding, and quality control that clinical trials apply to drug testing.

To minimize bias, you must randomize the presentation order. If Model A always appears on the left, you will conflate model quality with position preference. A proper A/B test randomly assigns positions per comparison. This keeps any position bias affects both models equally across the dataset. This randomization serves a important statistical purpose: it turns position effects from confounding variables into noise that averages to zero across the experiment. Without randomization, you cannot distinguish whether a 55% win rate for Model A shows real superiority or merely a 5% advantage from consistently appearing in the preferred position.

Equally important is anonymization of model identities. If annotators know they are comparing GPT-4 against an open-source model, their expectations and prior beliefs will contaminate their judgments. This is not mere speculation; experiments in wine tasting, music listening, and product evaluation consistently show that labels change perceived quality even when the underlying product is identical. In language model evaluation, the contamination risk is even higher because the community has strong prior beliefs about which providers produce better models. Blinded evaluation with randomly assigned model identifiers ensures that judgments reflect the actual response quality rather than brand reputation.

When comparing multiple models, use blocking to ensure balanced comparisons. If you have four models (A, B, C, D), you want roughly equal numbers of A-vs-B, A-vs-C, A-vs-D, B-vs-C, B-vs-D, and C-vs-D matchups. This prevents one model from being evaluated primarily against weak competitors, which would inflate its win rate. Blocking also ensures that the comparison graph remains connected, which is needed for global ranking estimation. In statistical terms, balanced blocking minimizes the variance of rating estimates by so that each model faces a representative sample of the competitor population rather than a biased subset.

The statistical validity of a comparison study depends on how many comparisons you collect and which prompts you use. A study that exclusively uses straightforward factual questions will give a misleadingly optimistic picture of models that excel at retrieval but struggle with reasoning. Conversely, a study dominated by creative writing prompts will favor expressive models over precise ones.

Stratified prompt sampling ensures that your evaluation covers the task distribution you care about. If your application involves 40% factual questions, 30% summarization, 20% creative writing, and 10% code generation, your prompt set should approximate this distribution. When you cannot specify the distribution in advance (because you are building a general assistant), you can sample prompts from real user conversations. This keeps the evaluation shows actual usage patterns rather than an idealized benchmark.

Prompt quality matters as much as prompt diversity. Ambiguous or poorly specified prompts produce noisy comparisons because different annotators interpret the task differently. A prompt like "write something about climate change" leaves so much to interpretation that two high-quality responses might be judged differently based solely on which annotator valued conciseness over comprehensiveness. Well-specified prompts with clear criteria reduce this variance and make comparisons more informative.

Even with perfect experimental design, annotation quality determines the floor on what your evaluation can reach. Annotators need explicit guidelines that specify what dimensions of quality to evaluate (helpfulness, accuracy, safety, fluency), how to handle edge cases (what to do when one response is better on some dimensions but worse on others), and how to calibrate their judgments (give examples of responses at different quality levels).

Calibration examples are particularly useful. Before beginning their session, annotators review a set of example pairs where the "correct" answer has been established by expert consensus. This anchors their judgment to the intended quality scale and reduces drift between annotators. Some evaluation platforms use periodic calibration checks throughout the annotation session, flagging annotators whose accuracy on calibration items degrades and triggering retraining or review.

Inter-annotator agreement (IAA) gives a quantitative measure of annotation consistency. For binary win/loss judgments, Cohen's kappa is the standard metric:

$$\kappa = \frac{p_o - p_e}{1 - p_e}$$

where:

• $\kappa$: Cohen's kappa, measuring agreement corrected for chance
• $p_o$: the observed agreement rate (fraction of comparisons where annotators agree)
• $p_e$: the expected agreement rate under random labeling (typically 0.5 for binary choices)

Values of $\kappa > 0.6$ are generally considered acceptable for preference evaluation. If $\kappa$ is lower, you should investigate whether guidelines are unclear, whether the prompts are inherently ambiguous, or whether some annotators are giving low-quality work. Multiple redundant annotations per comparison allow you to detect and filter unreliable annotators before they contaminate your dataset.

Not all comparisons yield clear winners. Annotators should have the option to declare a tie when responses are indistinguishable in quality. In Elo systems, ties typically award half a point to each competitor, but you can also model them explicitly using the Bradley-Terry model with ties, which extends the binary preference probability to a three-outcome case.

The probability that model $i$ ties with model $j$ can be modeled as:

$$P(\text{tie}) = \frac{\theta_{ij}}{1 + \theta_{ij} + \theta_{ji}}$$

where:

• $P(\text{tie})$: the probability that model $i$ ties with model $j$
• $\theta_{ij} = e^{(r_i - r_j)/\sigma}$: the relative strength parameter for model $i$ against model $j$, based on their rating difference
• $r_i, r_j$: the ratings of model $i$ and model $j$, respectively
• $\sigma$: the draw margin parameter controlling how likely ties are
• $\theta_{ji} = e^{(r_j - r_i)/\sigma}$: the inverse relative strength parameter for model $j$ against model $i$

The draw margin parameter $\sigma$ controls the zone of ambiguity where models are considered evenly matched. When rating differences are small relative to $\sigma$, ties become probable; when differences are large, ties become vanishingly unlikely. This formulation recognizes that ties are most likely when competitors are closely matched, whereas large skill gaps should produce decisive outcomes. However, most practical implementations treat ties as half-wins for simplicity, accepting a slight bias in exchange for computational tractability.

The decision of whether to allow ties also affects annotation behavior. Systems that force a winner on every comparison eliminate the cognitive option of "these are equivalent," which can reduce deliberation and push annotators toward gut reactions. Systems that allow ties risk annotators using the "tie" option as a way to avoid difficult decisions, inflating the tie rate beyond its true value. Monitoring the tie rate across annotators and flagging unusually high tie rates can help identify this pattern.