Inter-Annotator Agreement › Summary

Inter-annotator agreement turns the vague notion of "label quality" into quantifiable, comparable statistics that can be tracked across annotation rounds, compared across datasets, and used to make principled decisions about when data is ready for use. We have examined three complementary approaches, each suited to different annotation scenarios:

• Cohen's kappa measures agreement between two specific raters with potentially different biases, which makes it ideal for validating a primary annotator against an expert reviewer or for measuring consistency between two automated systems. It preserves rater-specific marginal distributions and is the standard choice when rater identity matters.
• Fleiss' kappa generalizes to any number of raters treated as interchangeable samples from a population, suitable for crowdsourcing scenarios where you care about the pool as a whole rather than individual workers. It uses pooled marginals and equals Scott's pi in the two-rater case.
• Krippendorff's alpha provides the most general framework, handling missing data, varying numbers of raters per item, and different measurement scales through configurable distance metrics. It is the preferred choice for complex annotation schemes with incomplete data or ordinal ratings.

All three coefficients share the chance-correction framework, comparing observed agreement against expected random agreement. This correction prevents inflated scores on imbalanced datasets but introduces sensitivity to prevalence and marginal distributions. The prevalence paradox and the bias problem are the most common pitfalls in practice, and both can lead to misleading conclusions if you report kappa values without understanding what drives them.

When agreement is low, we have options beyond simple majority voting. Soft labels preserve the uncertainty inherent in borderline cases. Iterative guideline refinement addresses systematic disagreement by clarifying ambiguous decision boundaries. Disagreement-weighted training reduces the influence of unreliable labels. The appropriate strategy depends on whether disagreement represents noise to be eliminated or signal to be preserved.

As we move toward evaluation methodologies that use LLMs as judges, the principles of chance-corrected agreement remain central. Validating an automated judge requires measuring its agreement with human judgments using the same statistical rigor we apply to human annotators. The next chapter on Preference Evaluation will explore how these agreement metrics apply specifically to the pairwise and ranking judgments that drive modern alignment techniques like RLHF and DPO, and how to design preference annotation pipelines that achieve both high IAA and valid coverage of the preference space.