Inter-Annotator Agreement › Handling Disagreement

High inter-annotator agreement validates our annotation scheme, but what happens when agreement is low? Disagreement is not always noise. Sometimes it signals ambiguous examples, underspecified guidelines, or subjective phenomena. Understanding and handling disagreement appropriately is as important as measuring it.

The simplest approach to resolving disagreement is adjudication: a third expert reviews cases where raters disagree and determines the "correct" label. This works well for objective tasks with clear gold standards (such as syntactic parsing or coreference resolution) but risks imposing artificial certainty on subjective phenomena. When a quality rater says a response is "helpful" and another says it is "not helpful," adjudication forces a binary choice that erases real ambiguity.

For multi-rater scenarios, majority voting selects the most common label. With two raters, this requires a tie-breaker rule. Majority voting preserves the most likely interpretation but discards information about uncertainty. If two annotators rate sentiment as Positive and Neutral while a third says Negative, majority voting yields Positive, but the disagreement suggests this might be a borderline case that a model should approach with calibrated uncertainty rather than confident classification.

Rather than forcing hard choices, we can preserve disagreement as soft labels. If three annotators label an item as [Positive, Positive, Neutral], the soft label becomes a probability distribution: $P(\text{Positive}) = 0.67$, $P(\text{Neutral}) = 0.33$.

Training models on soft labels rather than hard majority votes captures the nuance of borderline cases. Research on label smoothing and distribution matching has shown that models trained with soft targets can produce better-calibrated predictions. As we discussed in Part XXXVII: Alignment and RLHF, reward models are often trained on preference probabilities derived from multiple human judgments rather than binary choices. This approach acknowledges that some comparisons are close calls and that a model should assign similar scores to the competing responses.

The degree of annotation disagreement can itself be used as a signal. Items where annotators consistently disagree might indicate that the underlying construct is inherently ambiguous, or that the item sits at a decision boundary in the feature space. Both interpretations are informative for model design: the first suggests we should collect more annotations or refine the task definition, the second suggests the model should be allowed to output uncertainty.

Low agreement items might deserve different treatment during training. Items with high annotator disagreement can be:

• Weighted down in the loss function to reduce their influence on learned representations
• Excluded from training if disagreement exceeds a threshold, keeping only high-quality signal
• Flagged for guideline refinement when systematic disagreement patterns emerge
• Treated as belonging to a separate "ambiguous" class, giving the model an explicit way to express uncertainty

In evaluation, disagreement-aware metrics report both performance on high-agreement items (where labels are reliable) and performance on the full dataset. A model that performs well only on unambiguous items might be exploiting superficial features rather than understanding the task. Similarly, a benchmark where 30% of items have majority-vote accuracy below 60% among human raters should not be treated as having a clean gold standard.

Persistent disagreement often indicates guideline deficiencies. When annotators systematically disagree on entity boundaries in NER or on whether a statement constitutes a "hallucination," the annotation guidelines need clarification. Iterative annotation, where initial rounds inform guideline refinement, typically improves kappa scores in subsequent rounds.

However, be cautious of over-fitting guidelines to specific annotators. If you refine guidelines until two particular annotators agree, you may have simply matched the idiosyncrasies of those two people, not improved generalizability. Ideal guideline development involves diverse annotators, multiple annotation rounds, and explicit documentation of edge cases so that future annotators can apply the same standards.

A practical workflow for high-quality annotation projects looks like this. First, annotate a small pilot batch (100-200 items) and measure kappa. Second, analyze disagreement patterns to identify which item types, which category distinctions, or which annotator pairs are most problematic. Third, update guidelines to address specific sources of confusion. Fourth, annotate a second pilot batch and measure kappa again. Only once kappa exceeds your threshold should you proceed with the full dataset. This iterative approach front-loads cost but dramatically reduces the risk of discovering systematic quality problems after annotating tens of thousands of items.