Human Preference Data › Quality Control and Agreement

Preference data quality directly impacts alignment quality, and the relationship is not linear. Low-quality data is not just less efficient than high-quality data; it actively trains reward models in wrong directions. A reward model trained on data where annotators systematically prefer verbose responses will assign higher rewards to verbosity, and the resulting aligned model will be verbose even when conciseness would serve users better. Quality control is not optional overhead; it is core to alignment.

Multiple mechanisms help ensure data reliability, and they work best in combination. No single quality control mechanism is sufficient on its own. Inter-annotator agreement metrics tell you whether annotators are consistent but not whether they are consistently correct. Gold questions catch random clicking but not systematic biases. Statistical outlier detection flags individual annotators but cannot catch collective biases shared by all annotators in your pool. Effective quality control requires layered defenses.

Having multiple annotators label the same comparison allows measurement of agreement, which provides a direct proxy for data quality. If annotators frequently disagree about which response is better, the data is noisy. If they almost always agree, the comparisons may be too easy to provide useful training signal. The target range of 70-80% agreement reflects a sweet spot where comparisons are challenging enough to be informative but consistent enough to be reliable.

Raw agreement rate is the simplest metric: the percentage of comparisons where annotators chose the same response. For binary choices (ignoring ties), random agreement is 50%, so rates should substantially exceed this. A raw agreement rate of 65% means annotators are agreeing at a rate only slightly better than chance, which suggests the data is very noisy. A rate of 90% might suggest the comparisons are too easy.

Cohen's Kappa addresses the key limitation of raw agreement rate: it does not account for the possibility that annotators might agree purely by chance. If two annotators are each selecting randomly, they would still agree about half the time on binary choices. We need a metric that tells us how much better than chance our annotators are agreeing.

Cohen's Kappa measures inter-annotator agreement while accounting for the probability of chance agreement. The core insight is that we want to measure the agreement that goes beyond what we would expect from random labeling. To compute Kappa, we first determine what fraction of agreement is due to chance by considering the marginal distributions of annotator choices, then measure how much actual agreement exceeds that baseline.

Formally, if we let $p_o$ be the observed agreement and $p_e$ be the expected agreement by chance, Cohen's Kappa is:

$$\kappa = \frac{p_o - p_e}{1 - p_e}$$

where:

• $\kappa$: the Cohen's Kappa coefficient, ranging from negative infinity to 1
• $p_o$: the observed agreement (the proportion of items where annotators chose the same option)
• $p_e$: the expected agreement by chance (derived from the marginal distributions of annotator choices, computed as the sum over all labels of the product of each annotator's marginal probability of choosing that label)
• $p_o - p_e$: the agreement achieved beyond chance, the numerator of the correction
• $1 - p_e$: the maximum possible agreement beyond chance, used to normalize the metric to a fixed scale

To understand this formula, consider a concrete example. Suppose two annotators each label 100 comparisons. Annotator 1 prefers Response A in 60 cases and Response B in 40 cases. Annotator 2 prefers Response A in 55 cases and Response B in 45 cases. They agree in 70 cases ($p_o = 0.70$).

The expected chance agreement is computed from their marginal distributions. The probability that both choose A by chance is $0.60 \times 0.55 = 0.33$. The probability that both choose B by chance is $0.40 \times 0.45 = 0.18$. So $p_e = 0.33 + 0.18 = 0.51$.

Applying the formula: $\kappa = (0.70 - 0.51) / (1 - 0.51) = 0.19 / 0.49 \approx 0.39$, which falls in the "fair" agreement range despite a raw agreement rate of 70%. This illustrates why Cohen's Kappa can paint a more sobering picture than raw agreement: it corrects for the substantial chance agreement that exists in binary choice tasks.

The numerator $p_o - p_e$ represents how much agreement we observed beyond what chance would predict. The denominator $1 - p_e$ represents the maximum possible improvement over chance, which normalizes the metric. A value of 0 implies agreement no better than chance, while 1 implies perfect agreement. Negative values indicate agreement worse than chance, which might occur if annotators are systematically misunderstanding the task or applying opposite criteria.

Krippendorff's Alpha is a more flexible agreement measure that handles multiple annotators simultaneously and deals gracefully with missing data. Unlike Cohen's Kappa, which is defined for exactly two annotators, Krippendorff's Alpha generalizes to any number of annotators and accommodates the situation where different subsets of annotators label different items. This makes it more practical for large annotation projects where not every annotator labels every comparison. Alpha values are interpreted similarly to Kappa: above 0.8 indicates high reliability, 0.67-0.8 is acceptable for tentative conclusions, and below 0.67 is considered unreliable.

Typical preference datasets achieve 70-80% pairwise agreement, corresponding to Cohen's Kappa values in the moderate to substantial range. Perfect agreement is neither expected nor desirable: it would suggest the comparisons are so easy they provide little training signal. A Kappa of 0.5-0.7 is typically considered acceptable for preference data collection in practice.

Let's work through a concrete numerical example to make the Kappa calculation concrete. Suppose we have two annotators, each labeling the same 10 preference comparisons with choices A, B, or Tie.

Their labels are:

Step 1: Compute observed agreement $p_o$. The annotators agree on comparisons 1, 3, 4, 5, 7, 8, 9, for a total of 7 agreements out of 10. So $p_o = 7/10 = 0.70$.

Step 2: Compute marginal distributions. Annotator 1 chose A on items 1,2,4,6,7,9 (count 6) and B on items 3,5,8,10 (count 4). That count mistakenly treats item 6 as A, but it is a Tie. The corrected counts are: Annotator 1 chose A on items 1,2,4,7,9 (5 times), B on items 3,5,8,10 (4 times), and Tie on item 6 (1 time). Annotator 2 chose A on items 1,4,6,7,9 (5 times), B on items 2,3,5,8 (4 times), and Tie on item 10 (1 time).

Step 3: Compute expected chance agreement $p_e$. For each label, multiply the proportion chosen by Annotator 1 by the proportion chosen by Annotator 2, then sum:

• For label A: $(5/10) \times (5/10) = 0.25$
• For label B: $(4/10) \times (4/10) = 0.16$
• For label Tie: $(1/10) \times (1/10) = 0.01$
• $p_e = 0.25 + 0.16 + 0.01 = 0.42$

Step 4: Compute Kappa:

$$\kappa = \frac{p_o - p_e}{1 - p_e} = \frac{0.70 - 0.42}{1 - 0.42} = \frac{0.28}{0.58} \approx 0.48$$

This falls in the "moderate" agreement range, which is typical for preference annotation tasks involving subtle quality judgments. The raw agreement of 70% looked more impressive than the Kappa of 0.48 indicates, because a substantial fraction of that agreement is attributable to both annotators frequently choosing A regardless of the comparison.

When annotators disagree, understanding why is as important as measuring how much. Different types of disagreement call for different responses, and categorizing disagreements systematically guides guideline improvements.

Ambiguous prompts create disagreement when different annotators interpret the prompt differently. If annotators disagree about what a prompt is asking for, they may evaluate responses against different implicit standards and produce inconsistent preferences. This type of disagreement calls for prompt revision or clarification rather than annotator retraining.

Subjective criteria generate disagreement that reflects real diversity in preferences. Some quality dimensions, like formality of tone or level of detail, have no objectively correct answer. Annotators with different backgrounds or preferences may prefer different styles without either being wrong. Documenting and quantifying this type of disagreement helps calibrate expectations and may motivate explicitly stratifying annotators by relevant characteristics.

Criteria conflicts occur when annotators weigh quality dimensions differently. One annotator might prioritize helpfulness while another prioritizes harmlessness, leading to different preferences for the same pair. This type of disagreement often responds well to clearer guidelines that specify priority orderings.

Annotation errors include cases where annotators misread a response, were confused by the interface, or made a mechanical mistake. These errors are often identifiable through follow-up review and can sometimes be corrected retrospectively. Some annotation platforms allow annotators to flag their own uncertainty or request clarification on specific comparisons.

Systematically categorizing disagreements across a sample of disputed comparisons can reveal patterns that guide targeted improvements to guidelines, training, or comparison design. The analysis pays for itself when it prevents the same type of disagreement from repeating across thousands of future annotations.

Several mechanisms work together to maintain data quality throughout collection:

Gold questions are comparisons with known correct answers embedded in the annotation stream. For example, one response might be clearly helpful and the other clearly harmful, where any reasonable annotator should prefer the helpful response. Annotators who miss gold questions can be flagged for retraining, additional supervision, or removal from the project. Gold questions also provide a continuous quality signal: if gold question accuracy drops over time for an annotator, it may indicate fatigue or decreasing engagement.

Consistency checks show the same comparison twice (sometimes with response order flipped) and verify whether the annotator's choice is consistent. Inconsistency on the same comparison indicates noise: the annotator is not making a stable quality judgment but rather clicking somewhat randomly. A small rate of inconsistency is expected (humans are not perfectly consistent), but a high rate indicates either a very difficult comparison or an unreliable annotator.

Feedback loops involve regular communication between project managers and annotators to discuss difficult cases, clarify guidelines, and address systematic problems. Annotation is an iterative process, and the people doing the annotation often have valuable insights about where the guidelines are unclear or missing important cases. Creating channels for this feedback and acting on it improves quality throughout the project.

Statistical outlier detection identifies annotators whose judgments systematically differ from the majority. If one annotator consistently prefers the shorter response while all others prefer the longer response, or consistently prefers Response A regardless of prompt, these patterns may indicate systematic bias, confusion about the task, or gaming. Some divergence is expected and may even be desirable (you want diverse perspectives), but extreme outliers that are statistically inconsistent with the annotator pool warrant investigation.