Cohen, Fleiss & Krippendorff: IAA Metrics & Implementation

Every supervised learning system rests on labeled data, and labeled data rests on human judgment. But human judgment is neither infallible nor consistent. Two linguists reading the same sentence may disagree about whether it expresses sincere anger or mild frustration. Two medical text annotators may draw entity boundaries at different places. Two quality raters evaluating an LLM response may have completely different intuitions about what "helpful" means. When we train a model on these annotations or use them as benchmarks, we inherit whatever inconsistencies the annotation process contained.

This is not a theoretical concern. Datasets built without systematic agreement checking have been shown to contain systematic disagreements that bias model behavior in surprising ways. A model trained on sentiment data where one annotator was consistently stricter than others will learn to predict a blend of their standards rather than any coherent human judgment. An evaluation benchmark where raters frequently disagree will produce unstable leaderboard rankings that change more with the choice of raters than with the actual capability of the models being tested.

Inter-annotator agreement (IAA) provides the statistical machinery to quantify this reliability. It measures how consistently multiple annotators label the same data, correcting for the possibility that they might agree simply by chance. Without this correction, a dataset with 90% positive examples and only two labels might show 82% "agreement" even if annotators are guessing randomly. This chapter explores the mathematics of chance-corrected agreement, from Cohen's foundational kappa for pairs of annotators through Krippendorff's alpha, which generalizes to any number of raters, any measurement scale, and missing data. Along the way, we examine how to handle the disagreements that inevitably arise, when high agreement can mislead you, and how to choose the right metric for your annotation task.

As we discussed in Part VI: Sequence Labeling, named entity recognition and part-of-speech tagging require careful annotation protocols. The metrics we develop here tell us whether those protocols are sufficiently clear to produce reproducible labels. In Part XXXVII: Alignment and RLHF, we saw that human preference data drives reward modeling. Before using such data, we must verify that human raters agree on what constitutes a "better" response. We'll explore preference-specific evaluation in the next chapter on Preference Evaluation, but the foundations we build here apply universally across all annotation tasks.

Consider a simple binary classification task: determining whether a movie review is positive or negative. Two annotators label 100 reviews. They agree on 75 reviews and disagree on 25. Their raw agreement is 75%, which seems respectable. But what if 90 of those reviews are positive, a highly imbalanced dataset?

If both annotators simply guessed "positive" every time without reading a single review, they would agree on 90 reviews by chance alone. Their 75% observed agreement would be below chance expectation, showing systematic disagreement masked by class imbalance. Raw agreement percentages, often called "percentage of agreement" or $P_o$, fail to tell us anything useful without accounting for how much agreement chance alone would produce.

This prevalence problem becomes even more acute in real-world NLP tasks. Named entity recognition datasets often have a high proportion of "O" (outside) tokens compared to entity tokens. If 95% of tokens have no entity label, two annotators who both copy-paste the majority class would agree 90% of the time. A chance-corrected coefficient would expose this false agreement immediately.

The denominator $1 - P_e$ represents the maximum possible improvement over chance. If chance agreement is already high (say 0.9 due to class imbalance), there is little room for improvement, making high coefficient values difficult to achieve even with careful annotators. This explains why kappa coefficients sometimes show paradoxical behavior that we examine later. Understanding this denominator is key to understanding both the power and the limitations of chance-corrected metrics.

The logic behind chance-corrected agreement can be thought of as asking: "Given how frequently each annotator uses each label, what agreement would we expect if their decisions were statistically independent?" If annotator A uses the "positive" label 70% of the time and annotator B uses it 60% of the time, and they are operating independently, we would expect them to both say "positive" on $0.70 \times 0.60 = 0.42$ of items purely by coincidence. Any agreement beyond that reflects reliability in the annotation process.

Cohen's kappa ($\kappa$), introduced by Jacob Cohen in 1960, remains the most widely cited agreement statistic for exactly two raters working with categorical data. It assumes that the raters are distinct individuals with potentially different tendencies (one might be stricter or more lenient than the other), and it does not assume the categories are ordered. Virtually every annotation paper in NLP that involves two raters reports this coefficient.

The starting point for Cohen's kappa is the contingency table (often called the agreement matrix) that shows how often each rater assigned each category. For two raters and $k$ categories, this is a $k \times k$ matrix where cell $(i, j)$ shows how many items rater 1 assigned to category $i$ while rater 2 assigned them to category $j$.

Let $p_{ij}$ represent the proportion of items in cell $(i, j)$. The observed agreement $P_o$ is the sum of diagonal elements, since diagonal cells represent cases where both raters agreed:

where:

• $p_{ii}$: the proportion of items that both raters assign to category $i$ (the diagonal elements of the agreement matrix)
• $k$: the number of categories
• $P_o$: the total observed agreement (sum of diagonal proportions)

The expected agreement by chance $P_e$ is computed under the assumption that the two raters' decisions are statistically independent. If rater 1 assigns fraction $p_{i+}$ to category $i$, and rater 2 independently assigns fraction $p_{+i}$ to category $i$, then the expected fraction of items where both agree on category $i$ is simply the product of these marginals:

where:

• $p_{i+}$: the row marginal proportion (the proportion of items rater 1 assigns to category $i$)
• $p_{+i}$: the column marginal proportion (the proportion of items rater 2 assigns to category $i$)
• $P_e$: the expected agreement by chance, calculated as the sum of products of marginal proportions

This is simply the probability that two independent draws from the respective marginal distributions would land on the same category, summed over all categories.

Cohen's kappa then combines these into the familiar chance-corrected formula:

where:

• $\kappa$: Cohen's kappa coefficient (chance-corrected agreement ranging from $-1$ to $1$)
• $P_o$: the observed agreement proportion
• $P_e$: the expected agreement by chance
• $p_{ii}$: the proportion of agreement on category $i$
• $p_{i+}$, $p_{+i}$: the marginal proportions for category $i$

Cohen's kappa treats the raters as fixed entities. We are measuring agreement between these specific two people, not generalizing to a population of potential raters. This is the basic philosophical difference between Cohen's kappa and later metrics like Fleiss' kappa and Krippendorff's alpha: those metrics treat raters as interchangeable samples from some broader rater population.

The metric is symmetric: swapping rater 1 and rater 2 does not change the value. This makes it appropriate when there is no natural distinction between the two raters, such as two independent coders annotating the same corpus. When one rater is designated "gold standard" and the other is being evaluated, weighted agreement with the gold standard might be more informative.

Cohen's kappa also assumes that all disagreements are equally costly. If your categories are "negative," "neutral," and "positive," kappa penalizes a "negative" vs "positive" disagreement exactly as much as a "negative" vs "neutral" disagreement. This is appropriate for nominal scales but becomes problematic for ordinal data. Weighted kappa addresses this limitation by letting you to specify that some disagreements are worse than others.

When categories have a natural ordering, the gap between Cohen's standard kappa and what you care about can be substantial. Consider a 5-point toxicity scale ranging from 1 (completely benign) to 5 (highly toxic). Standard kappa treats a rater disagreement of 1 vs 2 the same as 1 vs 5, even though the latter represents a far more serious discrepancy for downstream use.

Weighted kappa introduces a penalty matrix $w_{ij}$ that specifies how much credit to give for each type of agreement or near-agreement. The weighted observed and expected agreement become:

and weighted kappa is:

where:

• $w_{ij}$: the agreement weight for the pair of labels $(i, j)$, ranging from 1 (full credit) to 0 (no credit)
• $P_o^w$: the weighted observed agreement
• $P_e^w$: the weighted expected agreement
• $\kappa_w$: the weighted kappa coefficient

Two common weighting schemes are linear weighting ($w_{ij} = 1 - \frac{|i-j|}{k-1}$) and quadratic weighting ($w_{ij} = 1 - \frac{(i-j)^2}{(k-1)^2}$). Quadratic weighting penalizes large disagreements disproportionately, which makes it the preferred choice when large errors are particularly harmful. Cohen's quadratic weighted kappa is identical to the intraclass correlation coefficient (ICC) under certain distributional assumptions, connecting it to the broader literature on reliability in psychology and medicine.

Landis and Koch (1977) proposed widely cited benchmarks for interpreting kappa values:

• $< 0.00$: Poor agreement
• $0.00 - 0.20$: Slight agreement
• $0.21 - 0.40$: Fair agreement
• $0.41 - 0.60$: Moderate agreement
• $0.61 - 0.80$: Substantial agreement
• $0.81 - 1.00$: Almost perfect agreement

However, these benchmarks face significant criticism and should not be applied mechanically. Kappa values depend heavily on prevalence and the difficulty of the task. A $\kappa$ of 0.6 might represent excellent agreement for subtle pragmatic phenomena like sarcasm detection, but poor agreement for clear-cut factual entity recognition where experienced annotators should achieve $\kappa > 0.9$. The benchmarks were derived empirically from medical studies and do not generalize automatically to NLP. Always interpret kappa values in the context of your task, your annotators' expertise, and the guidelines you provided.

A more principled approach is to set task-specific thresholds before annotation begins, based on your application requirements. If a model will be deployed in a high-stakes setting, you might require $\kappa > 0.8$ before accepting any annotation. If you are exploring a new task where guidelines are still being developed, $\kappa > 0.5$ with careful analysis of disagreement patterns might be acceptable.

Cohen's kappa does not generalize to more than two raters. When you have a crowdsourcing setup with ten workers or a research project where five domain experts each annotate the full corpus, you need a different approach. Fleiss' kappa (1971) extends the concept to any fixed number of raters $n \geq 2$, though it assumes the raters are interchangeable rather than distinct individuals.

This assumption of interchangeability is what distinguishes Fleiss' kappa from Cohen's. When you use Fleiss' kappa, you are implicitly treating your raters as random samples from a population of potential annotators, not specific individuals whose particular biases you care about. This makes it appropriate for crowdsourcing platforms like Amazon Mechanical Turk, where annotators are indeed sampled from a large pool.

Consider $N$ items and $k$ categories. For each item $i$, let $n_{ij}$ be the number of raters who assigned it to category $j$, where $\sum_{j=1}^{k} n_{ij} = n$ (each item gets exactly $n$ ratings). The proportion of raters assigning item $i$ to category $j$ is:

where:

• $n_{ij}$: the number of raters who assigned item $i$ to category $j$
• $n$: the total number of raters per item
• $p_{ij}$: the proportion of raters assigning item $i$ to category $j$

The observed agreement for item $i$ measures the extent to which raters agree on that specific item. Think of it as counting all pairs of raters who agreed and expressing this as a fraction of all possible rater pairs:

where:

• $P_i$: the extent of agreement among raters for item $i$ (ranging from 0 to 1)
• $n_{ij}$: the count of raters assigning item $i$ to category $j$
• $n$: the total number of raters
• $k$: the number of categories
• The first form counts agreeing pairs directly; the second simplifies computation using the sum of squares

To understand this formula, consider what happens at the extremes. If all $n$ raters choose the same category $j$, then $n_{ij} = n$ and $n_{ij}(n_{ij} - 1) = n(n-1)$. The sum equals $n(n-1)$ and $P_i = 1$. If raters split perfectly (each choosing a different category), then $n_{ij} = 1$ for each $j$ and $n_{ij}(n_{ij} - 1) = 0$ for all $j$, giving $P_i = 0$.

The mean observed agreement across all $N$ items is:

where:

• $P_o$: the mean observed agreement across all items
• $N$: the total number of items being rated
• $P_i$: the agreement score for item $i$

The chance agreement $P_e$ uses the proportion of all assignments falling into each category. This is a single pooled distribution across all raters, which reflects the interchangeability assumption:

where:

• $P_j$: the overall proportion of assignments to category $j$ across all items and raters
• $P_e$: the expected chance agreement (the sum of squared category proportions)

The intuition for $P_e = \sum_j P_j^2$ is clean: if we sampled two raters at random and both assigned labels independently from the pooled distribution, the probability they would pick the same category $j$ is $P_j \cdot P_j = P_j^2$. Summing over all categories gives the total chance agreement.

Fleiss' kappa is then:

where:

• $\kappa_F$: Fleiss' kappa coefficient
• $P_o$: the mean observed agreement across items
• $P_e$: the expected chance agreement based on category proportions

The most important practical difference between Fleiss' and Cohen's kappa is the choice of marginal distributions. Cohen's kappa uses separate marginals for each rater ($p_{i+}$ for rater 1 and $p_{+i}$ for rater 2). This reflects the fact that rater 1 might use "positive" 70% of the time while rater 2 uses it only 50% of the time. Fleiss' kappa uses a single pooled marginal ($P_j$), treating all raters as if they draw from the same distribution. This pooled approach is identical to what Scott's pi uses for two raters.

This has a subtle but important consequence. If you have exactly two raters with different biases and you apply Fleiss' kappa, you get Scott's pi rather than Cohen's kappa. The two coefficients can give substantially different values when rater biases differ substantially. When choosing between them, consider whether rater-specific tendencies are meaningful information (use Cohen's kappa) or noise to be averaged away (use Fleiss' kappa or Krippendorff's alpha).

When you have many raters or view raters as interchangeable samples from a larger population, Fleiss' kappa is the natural choice. In crowdsourcing research, it is particularly common because the identity of individual workers is less important than the overall reliability of the pool.

Krippendorff's alpha ($\alpha$) represents the most general agreement coefficient available. It accommodates:

• Any number of raters (varying per item if needed)
• Any number of categories
• Any measurement scale (nominal, ordinal, interval, ratio)
• Missing data, where some items receive fewer ratings than others

This flexibility makes it the preferred metric for complex annotation schemes in modern NLP, particularly when different items might have different numbers of ratings, when using ordinal scales like Likert items, or when the annotation project spans a long period where not all raters annotate all items. Klaus Krippendorff, who introduced the metric in 1970 and refined it extensively in subsequent decades, designed it explicitly for the messiness of real-world content analysis.

Krippendorff's alpha uses a disagreement-based formulation rather than an agreement-based one, but the underlying logic is identical to the chance-correction framework we have seen throughout this chapter:

where:

• $\alpha$: Krippendorff's alpha coefficient
• $D_o$: the observed disagreement among raters
• $D_e$: the expected disagreement by chance

When $D_o = 0$ (perfect agreement), $\alpha = 1$. When $D_o = D_e$ (agreement equals chance), $\alpha = 0$. When $D_o > D_e$ (worse than chance), $\alpha < 0$.

For nominal data (categories without order), the observed disagreement for an item is the proportion of rater pairs that disagree:

where:

• $D_o$: the observed disagreement for an item
• $n_c$: the count of raters choosing category $c$
• $n$: the total number of raters for that item
• The sum is taken over all categories $c$

The expected disagreement assumes a multinomial distribution based on the overall category frequencies across all data. If the probability of any given annotation being category $c$ is $\pi_c$, then the probability that two independent annotations disagree is:

where:

• $D_e$: the expected disagreement by chance
• $\pi_c$: the overall proportion of assignments to category $c$ across all data
• The term $\sum_{c} \pi_c^2$ represents the probability of chance agreement (the same formula as $P_e$ in Fleiss' kappa)

Notice that for nominal data with complete observations, Krippendorff's alpha and Fleiss' kappa give identical results. The differences between them emerge when data is missing or when using non-nominal measurement scales.

This is where Krippendorff's alpha truly distinguishes itself. Real annotation projects almost always have missing data. Annotators drop out, items are too difficult for some raters, or different items are assigned to different subsets of raters. Cohen's kappa and Fleiss' kappa require complete data for each item.

Krippendorff's alpha handles missing data through coincidence matrices rather than contingency tables. Instead of requiring every rater to label every item, we compute the observed disagreement from all pairs of observations that are present. For each item $i$ with $m_i \geq 2$ valid ratings, we consider all ordered pairs of raters and record which categories they chose. Items with fewer than 2 valid ratings are simply skipped.

For $m$ items with varying numbers of raters, we construct a coincidence matrix where cell $(c, k)$ contains the number of times any rater pair for any item assigned one to category $c$ and the other to category $k$. The diagonal contains agreements; off-diagonals contain disagreements. This matrix is symmetric, and each item with $n_i$ raters contributes $n_i(n_i - 1)$ entries.

The resulting coincidence matrix is a sufficient statistic for computing both $D_o$ and $D_e$, regardless of how many ratings each item has. This is the elegant mathematical property that lets alpha handle arbitrary missingness patterns without any special-casing.

For ordinal, interval, or ratio scales, not all disagreements are equal. A 5-star quality rater who gives 3 stars while their colleague gives 4 stars is closer to agreement than one who gives 1 star. Krippendorff's alpha incorporates a difference function $\delta_{ck}^2$ that weights disagreements by their distance:

where:

• $o_{ck}$: the observed coincidences between categories $c$ and $k$
• $e_{ck}$: the expected coincidences between categories $c$ and $k$
• $\delta_{ck}^2$: the squared distance between categories $c$ and $k$ (weighting function)
• The numerator sums observed disagreements weighted by distance; the denominator sums expected disagreements weighted by distance

Different measurement scales suggest different distance functions:

• Nominal: $\delta_{ck}^2 = 0$ if $c = k$, else $1$ (binary disagreement, all errors equal)
• Ordinal: $\delta_{ck}^2 = \left(\sum_{g=\min(c,k)}^{\max(c,k)} n_g - \frac{n_c + n_k}{2}\right)^2$ (rank-based distances)
• Interval: $\delta_{ck}^2 = (c - k)^2$ (squared difference in values)
• Ratio: $\delta_{ck}^2 = \left(\frac{c - k}{c + k}\right)^2$ (squared relative difference)

For Likert-scale ratings of LLM response quality, interval alpha is usually appropriate because the scale is treated as having equal spacing between levels. For judgments that have a natural zero point and where the ratio between values is meaningful (such as response latency in milliseconds), ratio alpha is more appropriate. The choice of distance function should be driven by the measurement theory underlying your scale, not by which choice makes the number look best.

Let's calculate all three metrics on a concrete example. Suppose three annotators (A, B, C) label 10 sentences for sentiment: Positive (P), Neutral (N), or Negative (G).

Six items (1, 3, 4, 6, 8, 9) are unanimously agreed upon. The four disagreement items (2, 5, 7, 10) each have a 2-1 split, with different categories causing trouble in each case.

First, construct the agreement matrix between A and B:

Observed agreement:

Row marginals for A: $p_{P+} = 0.5$, $p_{N+} = 0.3$, $p_{G+} = 0.2$ Column marginals for B: $p_{+P} = 0.6$, $p_{+N} = 0.3$, $p_{+G} = 0.1$

Expected agreement:

Cohen's kappa:

This falls in the "moderate" range. Notice that A uses "Negative" 20% of the time while B uses it only 10% of the time. This rater-level difference in base rates is exactly what Cohen's kappa captures through separate marginals.

For Fleiss' kappa, we calculate $P_i$ for each item:

• Item 1 (P,P,P):

• Item 2 (P,P,N):

• Item 3 (N,N,N): $1.0$

• Item 4 (P,P,P): $1.0$

• Item 5 (N,P,N):

• Item 6 (G,G,G): $1.0$

• Item 7 (P,N,P):

• Item 8 (N,N,N): $1.0$

• Item 9 (P,P,P): $1.0$

• Item 10 (G,P,G):

Mean observed agreement:

Category proportions (total assignments = 10 items $\times$ 3 raters = 30):

• $P_P = 15/30 = 0.5$
• $P_N = 10/30 = 0.333$
• $P_G = 5/30 = 0.167$

Expected agreement:

Fleiss' kappa:

For nominal data with 3 raters per item, we build the coincidence matrix. Each item contributes ordered pairs of ratings, one per ordered pair of raters.

Item 5: A(N), B(P), C(N). Ordered pairs: $(A,B):N$-$P$, $(A,C):N$-$N$, $(B,A):P$-$N$, $(B,C):P$-$N$, $(C,A):N$-$N$, $(C,B):N$-$P$. This contributes $o_{NP} += 2$, $o_{PN} += 2$, $o_{NN} += 2$.

Counting all contributions:

• $o_{PP}$: Items 1(6), 2(2), 4(6), 7(2), 9(6) $= 22$
• $o_{NN}$: Items 3(6), 5(2), 8(6) $= 14$
• $o_{GG}$: Items 6(6), 10(2) $= 8$
• $o_{PN} = o_{NP}$: Items 2(2), 5(2), 7(2) $= 6$ each
• $o_{PG} = o_{GP}$: Item 10(2) $= 2$ each
• $o_{NG} = o_{GN}$: None $= 0$

Total: $22 + 14 + 8 + 6 + 6 + 2 + 2 = 60$ (which matches $10 \times 3 \times 2 = 60$ ordered pairs).

Observed disagreement:

Expected disagreement:

Alpha:

For this case with complete data and nominal categories, Fleiss' kappa and Krippendorff's alpha give the same value (0.563), as theory predicts. Cohen's kappa between specific rater pairs varies. The A vs B pair showed $\kappa = 0.492$; A vs C and B vs C would produce different values because each pair has different marginal distributions.

Let's implement these calculations in Python, verifying our manual computation and showing library usage. The code also handles a missing-data scenario that illustrates Krippendorff's alpha's unique capabilities.

High inter-annotator agreement validates our annotation scheme, but what happens when agreement is low? Disagreement is not always noise. Sometimes it signals ambiguous examples, underspecified guidelines, or subjective phenomena. Understanding and handling disagreement appropriately is as important as measuring it.

The simplest approach to resolving disagreement is adjudication: a third expert reviews cases where raters disagree and determines the "correct" label. This works well for objective tasks with clear gold standards (such as syntactic parsing or coreference resolution) but risks imposing artificial certainty on subjective phenomena. When a quality rater says a response is "helpful" and another says it is "not helpful," adjudication forces a binary choice that erases real ambiguity.

For multi-rater scenarios, majority voting selects the most common label. With two raters, this requires a tie-breaker rule. Majority voting preserves the most likely interpretation but discards information about uncertainty. If two annotators rate sentiment as Positive and Neutral while a third says Negative, majority voting yields Positive, but the disagreement suggests this might be a borderline case that a model should approach with calibrated uncertainty rather than confident classification.

Rather than forcing hard choices, we can preserve disagreement as soft labels. If three annotators label an item as [Positive, Positive, Neutral], the soft label becomes a probability distribution: $P(\text{Positive}) = 0.67$, $P(\text{Neutral}) = 0.33$.

Training models on soft labels rather than hard majority votes captures the nuance of borderline cases. Research on label smoothing and distribution matching has shown that models trained with soft targets can produce better-calibrated predictions. As we discussed in Part XXXVII: Alignment and RLHF, reward models are often trained on preference probabilities derived from multiple human judgments rather than binary choices. This approach acknowledges that some comparisons are close calls and that a model should assign similar scores to the competing responses.

The degree of annotation disagreement can itself be used as a signal. Items where annotators consistently disagree might indicate that the underlying construct is inherently ambiguous, or that the item sits at a decision boundary in the feature space. Both interpretations are informative for model design: the first suggests we should collect more annotations or refine the task definition, the second suggests the model should be allowed to output uncertainty.

Low agreement items might deserve different treatment during training. Items with high annotator disagreement can be:

• Weighted down in the loss function to reduce their influence on learned representations
• Excluded from training if disagreement exceeds a threshold, keeping only high-quality signal
• Flagged for guideline refinement when systematic disagreement patterns emerge
• Treated as belonging to a separate "ambiguous" class, giving the model an explicit way to express uncertainty

In evaluation, disagreement-aware metrics report both performance on high-agreement items (where labels are reliable) and performance on the full dataset. A model that performs well only on unambiguous items might be exploiting superficial features rather than understanding the task. Similarly, a benchmark where 30% of items have majority-vote accuracy below 60% among human raters should not be treated as having a clean gold standard.

Persistent disagreement often indicates guideline deficiencies. When annotators systematically disagree on entity boundaries in NER or on whether a statement constitutes a "hallucination," the annotation guidelines need clarification. Iterative annotation, where initial rounds inform guideline refinement, typically improves kappa scores in subsequent rounds.

However, be cautious of over-fitting guidelines to specific annotators. If you refine guidelines until two particular annotators agree, you may have simply matched the idiosyncrasies of those two people, not improved generalizability. Ideal guideline development involves diverse annotators, multiple annotation rounds, and explicit documentation of edge cases so that future annotators can apply the same standards.

A practical workflow for high-quality annotation projects looks like this. First, annotate a small pilot batch (100-200 items) and measure kappa. Second, analyze disagreement patterns to identify which item types, which category distinctions, or which annotator pairs are most problematic. Third, update guidelines to address specific sources of confusion. Fourth, annotate a second pilot batch and measure kappa again. Only once kappa exceeds your threshold should you proceed with the full dataset. This iterative approach front-loads cost but dramatically reduces the risk of discovering systematic quality problems after annotating tens of thousands of items.

Chance-corrected agreement coefficients, while needed tools in the NLP practitioner's toolkit, suffer from well-documented paradoxes and limitations. Understanding these issues is not just academic: they determine which coefficient you should choose for a given task and how you should report and interpret your results.

When class distributions are highly skewed, kappa coefficients can be surprisingly low even with high raw agreement. Consider a medical diagnosis task where 95% of cases are "healthy." Two doctors might agree on 90% of cases (both saying healthy) but disagree on the 5% who are sick. Raw agreement is 90%, but if they randomly guessed healthy 95% of the time, chance agreement would be:

The kappa would be:

This result suggests negative agreement despite 90% accuracy. The paradox arises because chance agreement is already 90.5% given the class imbalance, leaving essentially no room for improvement. The kappa denominator $1 - P_e = 0.095$ amplifies any noise in the numerator.

The implication for NLP practice is significant. For sequence labeling tasks like NER, where most tokens are "outside" entities, kappa values will routinely look poor even for high-quality annotators. This prevalence paradox means kappa is unreliable for heavily imbalanced datasets. Some researchers recommend reporting both raw agreement and prevalence-adjusted bias-adjusted kappa (PABAK), though this metric loses the chance-correction interpretation. Others recommend reporting agreement separately for each class (macro-averaged kappa per category) to identify which specific categories are problematic.

Cohen's kappa assumes fixed marginals for each rater. When raters have different base rates (one is more lenient than another), kappa paradoxically decreases compared to Scott's pi, which pools marginals. Byrt, Bishop, and Carlin (1993) formalized this as the "bias" component of kappa's limitations: if rater A uses "positive" 80% of the time but rater B uses it 40% of the time, the pooled marginal approach assumes an intermediate rate, while Cohen's kappa computes chance agreement using these divergent marginals separately, creating a lower expected agreement.

Fleiss' kappa and Krippendorff's alpha use pooled marginals, effectively treating rater differences as noise to be averaged out. This makes them more stable but less sensitive to systematic rater biases. The choice between coefficients depends on whether you view rater differences as meaningful signal (annotator A is systematically more strict) or measurement error (raters should be interchangeable). In practice, when two specific experts consistently differ in their labeling tendencies, Cohen's kappa better captures the challenge of reconciling their perspectives.

Standard kappa statistics assume categorical data. Modern NLP increasingly uses continuous ratings (quality scores from 1-5) or rankings (preference pairs). While Krippendorff's alpha handles ordinal and interval scales through distance weighting, the interpretation becomes complex. A disagreement between ratings of 1 and 2 on a 5-point scale might mean something different than 4 vs 5, even if the numerical difference is identical: the bottom of the scale might be harder to distinguish perceptually than the middle.

For preference evaluation, which we will cover in the next chapter, we often use ranking agreement metrics like Kendall's $\tau$ or the Bradley-Terry model discussed in Part XXXVII: Alignment and RLHF. These capture the relative nature of preferences better than categorical agreement. A rater who consistently ranks response A above response B agrees with another rater who does the same, even if they would assign different absolute quality scores to each response.

High inter-annotator agreement does not guarantee validity. Annotators might agree consistently while being consistently wrong relative to some objective standard, or they might agree on superficial features while missing the deeper linguistic phenomenon we care about. Agreement validates the reproducibility of the measurement, not the correctness of the construct.

This distinction matters when using human judgments as training data for LLMs. In Part XXXVI: Instruction Tuning, we saw that instruction-following datasets rely on human judgments of response quality. High IAA ensures consistency in those judgments, but if the annotators share systematic biases (cultural, temporal, or ideological), the resulting model inherits those biases. A group of annotators who all come from the same demographic background might agree strongly with each other while being unrepresentative of the broader population the model serves. We will explore bias measurement and mitigation in Part LVIII: Bias and Fairness.

The appropriate framework is to view IAA as necessary but not sufficient for dataset quality. You need agreement to ensure the labels are consistent, and you need validity studies (correlation with external criteria, expert review, or task performance) to ensure the labels measure what you intend.

Despite the Landis and Koch benchmarks, context determines what constitutes "good" agreement. For objective tasks like part-of-speech tagging on news text, $\kappa > 0.8$ is achievable and expected. For subjective tasks like sentiment analysis of tweets or assessing whether an LLM response "shows empathy," $\kappa = 0.6$ might represent excellent agreement given the inherent ambiguity.

Some practitioners use $\kappa = 0.67$ as a minimum threshold for reliable data, while $\kappa > 0.8$ indicates data suitable for algorithmic training without additional review. Values below 0.4 suggest the annotation scheme needs substantial revision before proceeding to large-scale labeling. These are starting points, not rules: always justify your threshold relative to your task requirements and your downstream use case.

When reporting IAA in a paper or technical report, always include the raw agreement $P_o$ alongside the chance-corrected coefficient, note the number of raters and items, report how missing data was handled if applicable, and indicate which specific variant of each metric you computed (standard vs weighted kappa, nominal vs ordinal alpha). This transparency allows readers to assess the quality of the annotation even if they would have chosen different thresholds.

Inter-annotator agreement turns the vague notion of "label quality" into quantifiable, comparable statistics that can be tracked across annotation rounds, compared across datasets, and used to make principled decisions about when data is ready for use. We have examined three complementary approaches, each suited to different annotation scenarios:

• Cohen's kappa measures agreement between two specific raters with potentially different biases, which makes it ideal for validating a primary annotator against an expert reviewer or for measuring consistency between two automated systems. It preserves rater-specific marginal distributions and is the standard choice when rater identity matters.
• Fleiss' kappa generalizes to any number of raters treated as interchangeable samples from a population, suitable for crowdsourcing scenarios where you care about the pool as a whole rather than individual workers. It uses pooled marginals and equals Scott's pi in the two-rater case.
• Krippendorff's alpha provides the most general framework, handling missing data, varying numbers of raters per item, and different measurement scales through configurable distance metrics. It is the preferred choice for complex annotation schemes with incomplete data or ordinal ratings.

All three coefficients share the chance-correction framework, comparing observed agreement against expected random agreement. This correction prevents inflated scores on imbalanced datasets but introduces sensitivity to prevalence and marginal distributions. The prevalence paradox and the bias problem are the most common pitfalls in practice, and both can lead to misleading conclusions if you report kappa values without understanding what drives them.

When agreement is low, we have options beyond simple majority voting. Soft labels preserve the uncertainty inherent in borderline cases. Iterative guideline refinement addresses systematic disagreement by clarifying ambiguous decision boundaries. Disagreement-weighted training reduces the influence of unreliable labels. The appropriate strategy depends on whether disagreement represents noise to be eliminated or signal to be preserved.

As we move toward evaluation methodologies that use LLMs as judges, the principles of chance-corrected agreement remain central. Validating an automated judge requires measuring its agreement with human judgments using the same statistical rigor we apply to human annotators. The next chapter on Preference Evaluation will explore how these agreement metrics apply specifically to the pairwise and ranking judgments that drive modern alignment techniques like RLHF and DPO, and how to design preference annotation pipelines that achieve both high IAA and valid coverage of the preference space.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about inter-annotator agreement and chance-corrected reliability coefficients.