Inter-Annotator Agreement › Limitations and Impact

Chance-corrected agreement coefficients, while needed tools in the NLP practitioner's toolkit, suffer from well-documented paradoxes and limitations. Understanding these issues is not just academic: they determine which coefficient you should choose for a given task and how you should report and interpret your results.

When class distributions are highly skewed, kappa coefficients can be surprisingly low even with high raw agreement. Consider a medical diagnosis task where 95% of cases are "healthy." Two doctors might agree on 90% of cases (both saying healthy) but disagree on the 5% who are sick. Raw agreement is 90%, but if they randomly guessed healthy 95% of the time, chance agreement would be:

$$\begin{aligned} P_e &= 0.95^2 + 0.05^2 \\ &= 0.9025 + 0.0025 \\ &= 0.905 \end{aligned}$$

The kappa would be:

$$\begin{aligned} \kappa &= \frac{0.90 - 0.905}{1 - 0.905} \\ &= \frac{-0.005}{0.095} \\ &\approx -0.05 \end{aligned}$$

This result suggests negative agreement despite 90% accuracy. The paradox arises because chance agreement is already 90.5% given the class imbalance, leaving essentially no room for improvement. The kappa denominator $1 - P_e = 0.095$ amplifies any noise in the numerator.

The implication for NLP practice is significant. For sequence labeling tasks like NER, where most tokens are "outside" entities, kappa values will routinely look poor even for high-quality annotators. This prevalence paradox means kappa is unreliable for heavily imbalanced datasets. Some researchers recommend reporting both raw agreement and prevalence-adjusted bias-adjusted kappa (PABAK), though this metric loses the chance-correction interpretation. Others recommend reporting agreement separately for each class (macro-averaged kappa per category) to identify which specific categories are problematic.

Cohen's kappa assumes fixed marginals for each rater. When raters have different base rates (one is more lenient than another), kappa paradoxically decreases compared to Scott's pi, which pools marginals. Byrt, Bishop, and Carlin (1993) formalized this as the "bias" component of kappa's limitations: if rater A uses "positive" 80% of the time but rater B uses it 40% of the time, the pooled marginal approach assumes an intermediate rate, while Cohen's kappa computes chance agreement using these divergent marginals separately, creating a lower expected agreement.

Fleiss' kappa and Krippendorff's alpha use pooled marginals, effectively treating rater differences as noise to be averaged out. This makes them more stable but less sensitive to systematic rater biases. The choice between coefficients depends on whether you view rater differences as meaningful signal (annotator A is systematically more strict) or measurement error (raters should be interchangeable). In practice, when two specific experts consistently differ in their labeling tendencies, Cohen's kappa better captures the challenge of reconciling their perspectives.

Standard kappa statistics assume categorical data. Modern NLP increasingly uses continuous ratings (quality scores from 1-5) or rankings (preference pairs). While Krippendorff's alpha handles ordinal and interval scales through distance weighting, the interpretation becomes complex. A disagreement between ratings of 1 and 2 on a 5-point scale might mean something different than 4 vs 5, even if the numerical difference is identical: the bottom of the scale might be harder to distinguish perceptually than the middle.

For preference evaluation, which we will cover in the next chapter, we often use ranking agreement metrics like Kendall's $\tau$ or the Bradley-Terry model discussed in Part XXXVII: Alignment and RLHF. These capture the relative nature of preferences better than categorical agreement. A rater who consistently ranks response A above response B agrees with another rater who does the same, even if they would assign different absolute quality scores to each response.

High inter-annotator agreement does not guarantee validity. Annotators might agree consistently while being consistently wrong relative to some objective standard, or they might agree on superficial features while missing the deeper linguistic phenomenon we care about. Agreement validates the reproducibility of the measurement, not the correctness of the construct.

This distinction matters when using human judgments as training data for LLMs. In Part XXXVI: Instruction Tuning, we saw that instruction-following datasets rely on human judgments of response quality. High IAA ensures consistency in those judgments, but if the annotators share systematic biases (cultural, temporal, or ideological), the resulting model inherits those biases. A group of annotators who all come from the same demographic background might agree strongly with each other while being unrepresentative of the broader population the model serves. We will explore bias measurement and mitigation in Part LVIII: Bias and Fairness.

The appropriate framework is to view IAA as necessary but not sufficient for dataset quality. You need agreement to ensure the labels are consistent, and you need validity studies (correlation with external criteria, expert review, or task performance) to ensure the labels measure what you intend.

Despite the Landis and Koch benchmarks, context determines what constitutes "good" agreement. For objective tasks like part-of-speech tagging on news text, $\kappa > 0.8$ is achievable and expected. For subjective tasks like sentiment analysis of tweets or assessing whether an LLM response "shows empathy," $\kappa = 0.6$ might represent excellent agreement given the inherent ambiguity.

Some practitioners use $\kappa = 0.67$ as a minimum threshold for reliable data, while $\kappa > 0.8$ indicates data suitable for algorithmic training without additional review. Values below 0.4 suggest the annotation scheme needs substantial revision before proceeding to large-scale labeling. These are starting points, not rules: always justify your threshold relative to your task requirements and your downstream use case.

When reporting IAA in a paper or technical report, always include the raw agreement $P_o$ alongside the chance-corrected coefficient, note the number of raters and items, report how missing data was handled if applicable, and indicate which specific variant of each metric you computed (standard vs weighted kappa, nominal vs ordinal alpha). This transparency allows readers to assess the quality of the annotation even if they would have chosen different thresholds.