Inter-Annotator Agreement › Inter-Annotator Agreement

Every supervised learning system rests on labeled data, and labeled data rests on human judgment. But human judgment is neither infallible nor consistent. Two linguists reading the same sentence may disagree about whether it expresses sincere anger or mild frustration. Two medical text annotators may draw entity boundaries at different places. Two quality raters evaluating an LLM response may have completely different intuitions about what "helpful" means. When we train a model on these annotations or use them as benchmarks, we inherit whatever inconsistencies the annotation process contained.

This is not a theoretical concern. Datasets built without systematic agreement checking have been shown to contain systematic disagreements that bias model behavior in surprising ways. A model trained on sentiment data where one annotator was consistently stricter than others will learn to predict a blend of their standards rather than any coherent human judgment. An evaluation benchmark where raters frequently disagree will produce unstable leaderboard rankings that change more with the choice of raters than with the actual capability of the models being tested.

Inter-annotator agreement (IAA) provides the statistical machinery to quantify this reliability. It measures how consistently multiple annotators label the same data, correcting for the possibility that they might agree simply by chance. Without this correction, a dataset with 90% positive examples and only two labels might show 82% "agreement" even if annotators are guessing randomly. This chapter explores the mathematics of chance-corrected agreement, from Cohen's foundational kappa for pairs of annotators through Krippendorff's alpha, which generalizes to any number of raters, any measurement scale, and missing data. Along the way, we examine how to handle the disagreements that inevitably arise, when high agreement can mislead you, and how to choose the right metric for your annotation task.

As we discussed in Part VI: Sequence Labeling, named entity recognition and part-of-speech tagging require careful annotation protocols. The metrics we develop here tell us whether those protocols are sufficiently clear to produce reproducible labels. In Part XXXVII: Alignment and RLHF, we saw that human preference data drives reward modeling. Before using such data, we must verify that human raters agree on what constitutes a "better" response. We'll explore preference-specific evaluation in the next chapter on Preference Evaluation, but the foundations we build here apply universally across all annotation tasks.