Inter-Annotator Agreement › Fleiss' Kappa

Cohen's kappa does not generalize to more than two raters. When you have a crowdsourcing setup with ten workers or a research project where five domain experts each annotate the full corpus, you need a different approach. Fleiss' kappa (1971) extends the concept to any fixed number of raters $n \geq 2$, though it assumes the raters are interchangeable rather than distinct individuals.

This assumption of interchangeability is what distinguishes Fleiss' kappa from Cohen's. When you use Fleiss' kappa, you are implicitly treating your raters as random samples from a population of potential annotators, not specific individuals whose particular biases you care about. This makes it appropriate for crowdsourcing platforms like Amazon Mechanical Turk, where annotators are indeed sampled from a large pool.

Consider $N$ items and $k$ categories. For each item $i$, let $n_{ij}$ be the number of raters who assigned it to category $j$, where $\sum_{j=1}^{k} n_{ij} = n$ (each item gets exactly $n$ ratings). The proportion of raters assigning item $i$ to category $j$ is:

$$p_{ij} = \frac{n_{ij}}{n}$$

where:

• $n_{ij}$: the number of raters who assigned item $i$ to category $j$
• $n$: the total number of raters per item
• $p_{ij}$: the proportion of raters assigning item $i$ to category $j$

The observed agreement for item $i$ measures the extent to which raters agree on that specific item. Think of it as counting all pairs of raters who agreed and expressing this as a fraction of all possible rater pairs:

$$\begin{aligned} P_i &= \frac{1}{n(n-1)} \sum_{j=1}^{k} n_{ij}(n_{ij} - 1) \\ &= \frac{1}{n(n-1)} \left[ \left(\sum_{j=1}^{k} n_{ij}^2\right) - n \right] \end{aligned}$$

where:

• $P_i$: the extent of agreement among raters for item $i$ (ranging from 0 to 1)
• $n_{ij}$: the count of raters assigning item $i$ to category $j$
• $n$: the total number of raters
• $k$: the number of categories
• The first form counts agreeing pairs directly; the second simplifies computation using the sum of squares

To understand this formula, consider what happens at the extremes. If all $n$ raters choose the same category $j$, then $n_{ij} = n$ and $n_{ij}(n_{ij} - 1) = n(n-1)$. The sum equals $n(n-1)$ and $P_i = 1$. If raters split perfectly (each choosing a different category), then $n_{ij} = 1$ for each $j$ and $n_{ij}(n_{ij} - 1) = 0$ for all $j$, giving $P_i = 0$.

The mean observed agreement across all $N$ items is:

$$P_o = \frac{1}{N} \sum_{i=1}^{N} P_i$$

where:

• $P_o$: the mean observed agreement across all items
• $N$: the total number of items being rated
• $P_i$: the agreement score for item $i$

The chance agreement $P_e$ uses the proportion of all assignments falling into each category. This is a single pooled distribution across all raters, which reflects the interchangeability assumption:

$$P_j = \frac{1}{N \cdot n} \sum_{i=1}^{N} n_{ij}$$ $$P_e = \sum_{j=1}^{k} P_j^2$$

where:

• $P_j$: the overall proportion of assignments to category $j$ across all items and raters
• $P_e$: the expected chance agreement (the sum of squared category proportions)

The intuition for $P_e = \sum_j P_j^2$ is clean: if we sampled two raters at random and both assigned labels independently from the pooled distribution, the probability they would pick the same category $j$ is $P_j \cdot P_j = P_j^2$. Summing over all categories gives the total chance agreement.

Fleiss' kappa is then:

$$\kappa_F = \frac{P_o - P_e}{1 - P_e}$$

where:

• $\kappa_F$: Fleiss' kappa coefficient
• $P_o$: the mean observed agreement across items
• $P_e$: the expected chance agreement based on category proportions

The most important practical difference between Fleiss' and Cohen's kappa is the choice of marginal distributions. Cohen's kappa uses separate marginals for each rater ($p_{i+}$ for rater 1 and $p_{+i}$ for rater 2). This reflects the fact that rater 1 might use "positive" 70% of the time while rater 2 uses it only 50% of the time. Fleiss' kappa uses a single pooled marginal ($P_j$), treating all raters as if they draw from the same distribution. This pooled approach is identical to what Scott's pi uses for two raters.

This has a subtle but important consequence. If you have exactly two raters with different biases and you apply Fleiss' kappa, you get Scott's pi rather than Cohen's kappa. The two coefficients can give substantially different values when rater biases differ substantially. When choosing between them, consider whether rater-specific tendencies are meaningful information (use Cohen's kappa) or noise to be averaged away (use Fleiss' kappa or Krippendorff's alpha).

When you have many raters or view raters as interchangeable samples from a larger population, Fleiss' kappa is the natural choice. In crowdsourcing research, it is particularly common because the identity of individual workers is less important than the overall reliability of the pool.