Inter-Annotator Agreement › Worked Example

Let's calculate all three metrics on a concrete example. Suppose three annotators (A, B, C) label 10 sentences for sentiment: Positive (P), Neutral (N), or Negative (G).

Six items (1, 3, 4, 6, 8, 9) are unanimously agreed upon. The four disagreement items (2, 5, 7, 10) each have a 2-1 split, with different categories causing trouble in each case.

First, construct the agreement matrix between A and B:

Observed agreement:

$$\begin{aligned} P_o &= \frac{4 + 2 + 1}{10} \\ &= 0.7 \end{aligned}$$

Row marginals for A: $p_{P+} = 0.5$, $p_{N+} = 0.3$, $p_{G+} = 0.2$ Column marginals for B: $p_{+P} = 0.6$, $p_{+N} = 0.3$, $p_{+G} = 0.1$

Expected agreement:

$$\begin{aligned} P_e &= (0.5 \times 0.6) + (0.3 \times 0.3) + (0.2 \times 0.1) \\ &= 0.30 + 0.09 + 0.02 \\ &= 0.41 \end{aligned}$$

Cohen's kappa:

$$\begin{aligned} \kappa &= \frac{0.7 - 0.41}{1 - 0.41} \\ &= \frac{0.29}{0.59} \\ &\approx 0.492 \end{aligned}$$

This falls in the "moderate" range. Notice that A uses "Negative" 20% of the time while B uses it only 10% of the time. This rater-level difference in base rates is exactly what Cohen's kappa captures through separate marginals.

For Fleiss' kappa, we calculate $P_i$ for each item:

• Item 1 (P,P,P):

$$\begin{aligned} P_1 &= \frac{1}{3(2)}(3^2 - 3) \\ &= \frac{6}{6} \\ &= 1.0 \end{aligned}$$

• Item 2 (P,P,N):

$$\begin{aligned} P_2 &= \frac{1}{6}(2^2 + 1^2 + 0 - 3) \\ &= \frac{3}{6} \\ &= 0.333 \end{aligned}$$

• Item 3 (N,N,N): $1.0$

• Item 4 (P,P,P): $1.0$

• Item 5 (N,P,N):

$$\begin{aligned} P_5 &= \frac{1}{6}(1^2 + 2^2 - 3) \\ &= \frac{2}{6} \\ &= 0.333 \end{aligned}$$

• Item 6 (G,G,G): $1.0$

• Item 7 (P,N,P):

$$\begin{aligned} P_7 &= \frac{1}{6}(2^2 + 1^2 - 3) \\ &= 0.333 \end{aligned}$$

• Item 8 (N,N,N): $1.0$

• Item 9 (P,P,P): $1.0$

• Item 10 (G,P,G):

$$\begin{aligned} P_{10} &= \frac{1}{6}(1^2 + 2^2 - 3) \\ &= 0.333 \end{aligned}$$

Mean observed agreement:

$$\begin{aligned} P_o &= \frac{1}{10}(1 + 0.333 + 1 + 1 + 0.333 + 1 + 0.333 + 1 + 1 + 0.333) \\ &= \frac{7.332}{10} \\ &= 0.733 \end{aligned}$$

Category proportions (total assignments = 10 items $\times$ 3 raters = 30):

• $P_P = 15/30 = 0.5$
• $P_N = 10/30 = 0.333$
• $P_G = 5/30 = 0.167$

Expected agreement:

$$\begin{aligned} P_e &= 0.5^2 + 0.333^2 + 0.167^2 \\ &= 0.25 + 0.111 + 0.028 \\ &= 0.389 \end{aligned}$$

Fleiss' kappa:

$$\begin{aligned} \kappa_F &= \frac{0.733 - 0.389}{1 - 0.389} \\ &= \frac{0.344}{0.611} \\ &\approx 0.563 \end{aligned}$$

For nominal data with 3 raters per item, we build the coincidence matrix. Each item contributes ordered pairs of ratings, one per ordered pair of raters.

Item 5: A(N), B(P), C(N). Ordered pairs: $(A,B):N$-$P$, $(A,C):N$-$N$, $(B,A):P$-$N$, $(B,C):P$-$N$, $(C,A):N$-$N$, $(C,B):N$-$P$. This contributes $o_{NP} += 2$, $o_{PN} += 2$, $o_{NN} += 2$.

Counting all contributions:

• $o_{PP}$: Items 1(6), 2(2), 4(6), 7(2), 9(6) $= 22$
• $o_{NN}$: Items 3(6), 5(2), 8(6) $= 14$
• $o_{GG}$: Items 6(6), 10(2) $= 8$
• $o_{PN} = o_{NP}$: Items 2(2), 5(2), 7(2) $= 6$ each
• $o_{PG} = o_{GP}$: Item 10(2) $= 2$ each
• $o_{NG} = o_{GN}$: None $= 0$

Total: $22 + 14 + 8 + 6 + 6 + 2 + 2 = 60$ (which matches $10 \times 3 \times 2 = 60$ ordered pairs).

Observed disagreement:

$$\begin{aligned} D_o &= \frac{o_{PN} + o_{NP} + o_{PG} + o_{GP}}{60} \\ &= \frac{6 + 6 + 2 + 2}{60} \\ &= \frac{16}{60} \\ &= 0.267 \end{aligned}$$

Expected disagreement:

$$\begin{aligned} D_e &= 1 - (P_P^2 + P_N^2 + P_G^2) \\ &= 1 - (0.5^2 + 0.333^2 + 0.167^2) \\ &= 1 - 0.389 \\ &= 0.611 \end{aligned}$$

Alpha:

$$\begin{aligned} \alpha &= 1 - \frac{0.267}{0.611} \\ &= 1 - 0.437 \\ &= 0.563 \end{aligned}$$

For this case with complete data and nominal categories, Fleiss' kappa and Krippendorff's alpha give the same value (0.563), as theory predicts. Cohen's kappa between specific rater pairs varies. The A vs B pair showed $\kappa = 0.492$; A vs C and B vs C would produce different values because each pair has different marginal distributions.