Inter-Annotator Agreement › Cohen's Kappa

Cohen's kappa ($\kappa$), introduced by Jacob Cohen in 1960, remains the most widely cited agreement statistic for exactly two raters working with categorical data. It assumes that the raters are distinct individuals with potentially different tendencies (one might be stricter or more lenient than the other), and it does not assume the categories are ordered. Virtually every annotation paper in NLP that involves two raters reports this coefficient.

The starting point for Cohen's kappa is the contingency table (often called the agreement matrix) that shows how often each rater assigned each category. For two raters and $k$ categories, this is a $k \times k$ matrix where cell $(i, j)$ shows how many items rater 1 assigned to category $i$ while rater 2 assigned them to category $j$.

Let $p_{ij}$ represent the proportion of items in cell $(i, j)$. The observed agreement $P_o$ is the sum of diagonal elements, since diagonal cells represent cases where both raters agreed:

$$P_o = \sum_{i=1}^{k} p_{ii}$$

where:

• $p_{ii}$: the proportion of items that both raters assign to category $i$ (the diagonal elements of the agreement matrix)
• $k$: the number of categories
• $P_o$: the total observed agreement (sum of diagonal proportions)

The expected agreement by chance $P_e$ is computed under the assumption that the two raters' decisions are statistically independent. If rater 1 assigns fraction $p_{i+}$ to category $i$, and rater 2 independently assigns fraction $p_{+i}$ to category $i$, then the expected fraction of items where both agree on category $i$ is simply the product of these marginals:

$$P_e = \sum_{i=1}^{k} p_{i+} \cdot p_{+i}$$

where:

• $p_{i+}$: the row marginal proportion (the proportion of items rater 1 assigns to category $i$)
• $p_{+i}$: the column marginal proportion (the proportion of items rater 2 assigns to category $i$)
• $P_e$: the expected agreement by chance, calculated as the sum of products of marginal proportions

This is simply the probability that two independent draws from the respective marginal distributions would land on the same category, summed over all categories.

Cohen's kappa then combines these into the familiar chance-corrected formula:

$$\begin{aligned} \kappa &= \frac{P_o - P_e}{1 - P_e} \\ &= \frac{\sum_{i} p_{ii} - \sum_{i} p_{i+} p_{+i}}{1 - \sum_{i} p_{i+} p_{+i}} \end{aligned}$$

where:

• $\kappa$: Cohen's kappa coefficient (chance-corrected agreement ranging from $-1$ to $1$)
• $P_o$: the observed agreement proportion
• $P_e$: the expected agreement by chance
• $p_{ii}$: the proportion of agreement on category $i$
• $p_{i+}$, $p_{+i}$: the marginal proportions for category $i$

Cohen's kappa treats the raters as fixed entities. We are measuring agreement between these specific two people, not generalizing to a population of potential raters. This is the basic philosophical difference between Cohen's kappa and later metrics like Fleiss' kappa and Krippendorff's alpha: those metrics treat raters as interchangeable samples from some broader rater population.

The metric is symmetric: swapping rater 1 and rater 2 does not change the value. This makes it appropriate when there is no natural distinction between the two raters, such as two independent coders annotating the same corpus. When one rater is designated "gold standard" and the other is being evaluated, weighted agreement with the gold standard might be more informative.

Cohen's kappa also assumes that all disagreements are equally costly. If your categories are "negative," "neutral," and "positive," kappa penalizes a "negative" vs "positive" disagreement exactly as much as a "negative" vs "neutral" disagreement. This is appropriate for nominal scales but becomes problematic for ordinal data. Weighted kappa addresses this limitation by letting you to specify that some disagreements are worse than others.

When categories have a natural ordering, the gap between Cohen's standard kappa and what you care about can be substantial. Consider a 5-point toxicity scale ranging from 1 (completely benign) to 5 (highly toxic). Standard kappa treats a rater disagreement of 1 vs 2 the same as 1 vs 5, even though the latter represents a far more serious discrepancy for downstream use.

Weighted kappa introduces a penalty matrix $w_{ij}$ that specifies how much credit to give for each type of agreement or near-agreement. The weighted observed and expected agreement become:

$$P_o^w = \sum_{i=1}^{k} \sum_{j=1}^{k} w_{ij} \cdot p_{ij}$$ $$P_e^w = \sum_{i=1}^{k} \sum_{j=1}^{k} w_{ij} \cdot p_{i+} \cdot p_{+j}$$

and weighted kappa is:

$$\kappa_w = \frac{P_o^w - P_e^w}{1 - P_e^w}$$

where:

• $w_{ij}$: the agreement weight for the pair of labels $(i, j)$, ranging from 1 (full credit) to 0 (no credit)
• $P_o^w$: the weighted observed agreement
• $P_e^w$: the weighted expected agreement
• $\kappa_w$: the weighted kappa coefficient

Two common weighting schemes are linear weighting ($w_{ij} = 1 - \frac{|i-j|}{k-1}$) and quadratic weighting ($w_{ij} = 1 - \frac{(i-j)^2}{(k-1)^2}$). Quadratic weighting penalizes large disagreements disproportionately, which makes it the preferred choice when large errors are particularly harmful. Cohen's quadratic weighted kappa is identical to the intraclass correlation coefficient (ICC) under certain distributional assumptions, connecting it to the broader literature on reliability in psychology and medicine.

Landis and Koch (1977) proposed widely cited benchmarks for interpreting kappa values:

• $< 0.00$: Poor agreement
• $0.00 - 0.20$: Slight agreement
• $0.21 - 0.40$: Fair agreement
• $0.41 - 0.60$: Moderate agreement
• $0.61 - 0.80$: Substantial agreement
• $0.81 - 1.00$: Almost perfect agreement

However, these benchmarks face significant criticism and should not be applied mechanically. Kappa values depend heavily on prevalence and the difficulty of the task. A $\kappa$ of 0.6 might represent excellent agreement for subtle pragmatic phenomena like sarcasm detection, but poor agreement for clear-cut factual entity recognition where experienced annotators should achieve $\kappa > 0.9$. The benchmarks were derived empirically from medical studies and do not generalize automatically to NLP. Always interpret kappa values in the context of your task, your annotators' expertise, and the guidelines you provided.

A more principled approach is to set task-specific thresholds before annotation begins, based on your application requirements. If a model will be deployed in a high-stakes setting, you might require $\kappa > 0.8$ before accepting any annotation. If you are exploring a new task where guidelines are still being developed, $\kappa > 0.5$ with careful analysis of disagreement patterns might be acceptable.