Inter-Annotator Agreement › Krippendorff's Alpha

Krippendorff's alpha ($\alpha$) represents the most general agreement coefficient available. It accommodates:

• Any number of raters (varying per item if needed)
• Any number of categories
• Any measurement scale (nominal, ordinal, interval, ratio)
• Missing data, where some items receive fewer ratings than others

This flexibility makes it the preferred metric for complex annotation schemes in modern NLP, particularly when different items might have different numbers of ratings, when using ordinal scales like Likert items, or when the annotation project spans a long period where not all raters annotate all items. Klaus Krippendorff, who introduced the metric in 1970 and refined it extensively in subsequent decades, designed it explicitly for the messiness of real-world content analysis.

Krippendorff's alpha uses a disagreement-based formulation rather than an agreement-based one, but the underlying logic is identical to the chance-correction framework we have seen throughout this chapter:

$$\alpha = 1 - \frac{D_o}{D_e}$$

where:

• $\alpha$: Krippendorff's alpha coefficient
• $D_o$: the observed disagreement among raters
• $D_e$: the expected disagreement by chance

When $D_o = 0$ (perfect agreement), $\alpha = 1$. When $D_o = D_e$ (agreement equals chance), $\alpha = 0$. When $D_o > D_e$ (worse than chance), $\alpha < 0$.

For nominal data (categories without order), the observed disagreement for an item is the proportion of rater pairs that disagree:

$$D_o = \frac{1}{n(n-1)} \sum_{c} n_c (n - n_c)$$

where:

• $D_o$: the observed disagreement for an item
• $n_c$: the count of raters choosing category $c$
• $n$: the total number of raters for that item
• The sum is taken over all categories $c$

The expected disagreement assumes a multinomial distribution based on the overall category frequencies across all data. If the probability of any given annotation being category $c$ is $\pi_c$, then the probability that two independent annotations disagree is:

$$D_e = 1 - \sum_{c} \pi_c^2$$

where:

• $D_e$: the expected disagreement by chance
• $\pi_c$: the overall proportion of assignments to category $c$ across all data
• The term $\sum_{c} \pi_c^2$ represents the probability of chance agreement (the same formula as $P_e$ in Fleiss' kappa)

Notice that for nominal data with complete observations, Krippendorff's alpha and Fleiss' kappa give identical results. The differences between them emerge when data is missing or when using non-nominal measurement scales.

This is where Krippendorff's alpha truly distinguishes itself. Real annotation projects almost always have missing data. Annotators drop out, items are too difficult for some raters, or different items are assigned to different subsets of raters. Cohen's kappa and Fleiss' kappa require complete data for each item.

Krippendorff's alpha handles missing data through coincidence matrices rather than contingency tables. Instead of requiring every rater to label every item, we compute the observed disagreement from all pairs of observations that are present. For each item $i$ with $m_i \geq 2$ valid ratings, we consider all ordered pairs of raters and record which categories they chose. Items with fewer than 2 valid ratings are simply skipped.

For $m$ items with varying numbers of raters, we construct a coincidence matrix where cell $(c, k)$ contains the number of times any rater pair for any item assigned one to category $c$ and the other to category $k$. The diagonal contains agreements; off-diagonals contain disagreements. This matrix is symmetric, and each item with $n_i$ raters contributes $n_i(n_i - 1)$ entries.

The resulting coincidence matrix is a sufficient statistic for computing both $D_o$ and $D_e$, regardless of how many ratings each item has. This is the elegant mathematical property that lets alpha handle arbitrary missingness patterns without any special-casing.

For ordinal, interval, or ratio scales, not all disagreements are equal. A 5-star quality rater who gives 3 stars while their colleague gives 4 stars is closer to agreement than one who gives 1 star. Krippendorff's alpha incorporates a difference function $\delta_{ck}^2$ that weights disagreements by their distance:

$$\alpha = 1 - \frac{\sum_{c,k} o_{ck} \delta_{ck}^2}{\sum_{c,k} e_{ck} \delta_{ck}^2}$$

where:

• $o_{ck}$: the observed coincidences between categories $c$ and $k$
• $e_{ck}$: the expected coincidences between categories $c$ and $k$
• $\delta_{ck}^2$: the squared distance between categories $c$ and $k$ (weighting function)
• The numerator sums observed disagreements weighted by distance; the denominator sums expected disagreements weighted by distance

Different measurement scales suggest different distance functions:

• Nominal: $\delta_{ck}^2 = 0$ if $c = k$, else $1$ (binary disagreement, all errors equal)
• Ordinal: $\delta_{ck}^2 = \left(\sum_{g=\min(c,k)}^{\max(c,k)} n_g - \frac{n_c + n_k}{2}\right)^2$ (rank-based distances)
• Interval: $\delta_{ck}^2 = (c - k)^2$ (squared difference in values)
• Ratio: $\delta_{ck}^2 = \left(\frac{c - k}{c + k}\right)^2$ (squared relative difference)

For Likert-scale ratings of LLM response quality, interval alpha is usually appropriate because the scale is treated as having equal spacing between levels. For judgments that have a natural zero point and where the ratio between values is meaningful (such as response latency in milliseconds), ratio alpha is more appropriate. The choice of distance function should be driven by the measurement theory underlying your scale, not by which choice makes the number look best.