Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Inter-Annotator Agreement › Code Implementation

Michael BrenndoerferMarch 8, 202611 min read

Language AI HandbookAll chapters

Part 55 of 67Human and Model Evaluation

Chapter 424Inter-Annotator Agreement › Code Implementation

Reading section 7 of 11 · Code Implementation · 11 minSection 7 of 11 · 11 min

Section 7 of 11 · 11 min read

Inter-Annotator Agreement › Code ImplementationOne section · Sections in this chapter7 / 11 ↓

Sections in this chapter11

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let's implement these calculations in Python, verifying our manual computation and showing library usage. The code also handles a missing-data scenario that illustrates Krippendorff's alpha's unique capabilities.

In[4]:

Code

import numpy as np

# Define our annotation data: 10 items, 3 annotators (A, B, C)
# Categories: 0=Negative(G), 1=Neutral(N), 2=Positive(P)
annotations = np.array(
    [
        [2, 2, 2],  # Item 1: P,P,P
        [2, 2, 1],  # Item 2: P,P,N
        [1, 1, 1],  # Item 3: N,N,N
        [2, 2, 2],  # Item 4: P,P,P
        [1, 2, 1],  # Item 5: N,P,N
        [0, 0, 0],  # Item 6: G,G,G
        [2, 1, 2],  # Item 7: P,N,P
        [1, 1, 1],  # Item 8: N,N,N
        [2, 2, 2],  # Item 9: P,P,P
        [0, 2, 0],  # Item 10: G,P,G
    ]
)

n_items, n_raters = annotations.shape
categories = np.unique(annotations)
k = len(categories)

Now we calculate Cohen's kappa between annotators A (column 0) and B (column 1).

In[5]:

Code

# Cohen's Kappa between A and B
rater_a = annotations[:, 0]
rater_b = annotations[:, 1]

# Using sklearn
kappa_sklearn = cohen_kappa_score(rater_a, rater_b)


# Manual calculation for verification
def cohens_kappa_manual(rater1, rater2, categories):
    n = len(rater1)
    # Agreement matrix
    agreement = np.zeros((len(categories), len(categories)))
    for i in range(n):
        idx1 = np.where(categories == rater1[i])[0][0]
        idx2 = np.where(categories == rater2[i])[0][0]
        agreement[idx1, idx2] += 1

    agreement = agreement / n  # Convert to proportions

    # Observed agreement
    p_o = np.trace(agreement)

    # Expected agreement (marginals)
    p_i_plus = np.sum(agreement, axis=1)
    p_plus_j = np.sum(agreement, axis=0)
    p_e = np.sum(p_i_plus * p_plus_j)

    kappa = (p_o - p_e) / (1 - p_e)
    return kappa, p_o, p_e


kappa_manual, p_o, p_e = cohens_kappa_manual(rater_a, rater_b, categories)

Out[6]:

Console

Cohen's Kappa (A vs B) using sklearn: 0.492
Manual calculation: kappa = (0.70 - 0.41) / (1 - 0.41) = 0.492

The manual calculation confirms our earlier arithmetic, showing moderate agreement between annotators A and B.

Out[7]:

Visualization

3x3 heatmap agreement matrix for annotators A and B on negative, neutral, and positive sentiment. — Agreement matrix between annotators A and B showing the count of label assignments in each category pair. Diagonal cells (upper-left to lower-right) represent items where both raters agreed. The three off-diagonal disagreements are spread evenly across category boundaries, with no single category pair accounting for all errors.

Next, we implement Fleiss' kappa for all three raters.

In[8]:

Code

def fleiss_kappa(annotations, categories):
    """
    Calculate Fleiss' kappa for multiple raters.
    annotations: n_items x n_raters array
    """
    n_items, n_raters = annotations.shape
    k = len(categories)

    # Count matrix: n_items x k
    count_matrix = np.zeros((n_items, k))
    for i in range(n_items):
        for j in range(n_raters):
            cat_idx = np.where(categories == annotations[i, j])[0][0]
            count_matrix[i, cat_idx] += 1

    # P_i for each item: proportion of agreeing ordered rater pairs
    P_i = (np.sum(count_matrix**2, axis=1) - n_raters) / (
        n_raters * (n_raters - 1)
    )
    P_bar = np.mean(P_i)

    # Overall proportion of assignments to each category
    P_j = np.sum(count_matrix, axis=0) / (n_items * n_raters)

    # Expected agreement
    P_e = np.sum(P_j**2)

    kappa = (P_bar - P_e) / (1 - P_e) if (1 - P_e) != 0 else 0

    return kappa, P_bar, P_e, P_j, P_i, count_matrix


kappa_fleiss, p_bar, p_e_fleiss, cat_props, P_i_values, count_mat = (
    fleiss_kappa(annotations, categories)
)

Out[9]:

Console

Fleiss' Kappa: 0.564
Observed agreement (P_bar): 0.733
Expected agreement (P_e): 0.389
Category proportions: {'G': '0.167', 'N': '0.333', 'P': '0.500'}

Fleiss' kappa shows moderate agreement across all three raters, slightly higher than the pairwise Cohen's kappa between A and B. Pooling all three raters raises the observed agreement while producing a slightly lower chance-agreement baseline for this dataset.

Out[10]:

Visualization

Bar chart of per-item Fleiss kappa agreement scores, with unanimous items in teal and disagreement items in blue. — Per-item agreement scores ($P_i$) for the Fleiss kappa calculation. Items 1, 3, 4, 6, 8, and 9 show perfect unanimous agreement ($P_i = 1.0$, shown in teal), while the four disagreement items (2, 5, 7, 10) each score 0.33, corresponding to a 2-vs-1 split among the three raters. The horizontal dashed line marks the mean observed agreement $P_o = 0.733$.

Now we implement Krippendorff's alpha, which requires building the coincidence matrix.

In[11]:

Code

def krippendorff_alpha(annotations, categories, metric="nominal"):
    """
    Calculate Krippendorff's alpha.
    Handles varying numbers of raters per item (missing data via NaN).
    """
    n_items, n_raters = annotations.shape
    k = len(categories)

    # Build coincidence matrix
    coincidence = np.zeros((k, k))

    for i in range(n_items):
        # Get valid annotations for this item (non-NaN if we had missing data)
        valid = annotations[i, :]
        n_valid = len(valid)

        # All ordered pairs of raters
        for r1, r2 in itertools.product(range(n_valid), range(n_valid)):
            if r1 != r2:
                c1 = np.where(categories == valid[r1])[0][0]
                c2 = np.where(categories == valid[r2])[0][0]
                coincidence[c1, c2] += 1

    total_coincidences = np.sum(coincidence)

    # Observed disagreement (for nominal: proportion of off-diagonal)
    if metric == "nominal":
        D_o = (total_coincidences - np.trace(coincidence)) / total_coincidences
    else:
        raise NotImplementedError("Only nominal metric implemented here")

    # Expected disagreement based on marginals
    category_counts = np.sum(coincidence, axis=1)
    n_total = np.sum(category_counts)
    probs = category_counts / n_total

    D_e = 1 - np.sum(probs**2)

    alpha = 1 - (D_o / D_e) if D_e != 0 else 0

    return alpha, D_o, D_e, coincidence


alpha, d_o, d_e, coin_mat = krippendorff_alpha(annotations, categories)

Out[12]:

Console

Krippendorff's Alpha: 0.564
Observed disagreement (D_o): 0.267
Expected disagreement (D_e): 0.611

Krippendorff's alpha matches Fleiss' kappa in this case because we have complete data, fixed numbers of raters, and nominal categories. The advantage of alpha becomes apparent with missing data or ordinal scales.

Out[13]:

Visualization

3x3 coincidence matrix heatmap for Krippendorff alpha showing diagonal agreements and off-diagonal disagreements in orange. — Coincidence matrix for Krippendorff's alpha, showing the frequency of ordered rater-pair label combinations across all items. Diagonal elements represent agreements between rater pairs. The dominant off-diagonal entries (P-N and N-P) indicate that positive-vs-neutral confusion accounts for most disagreements, while positive-negative confusion is rare.

Out[14]:

Visualization

Bar chart comparing Cohen kappa, Fleiss kappa, and Krippendorff alpha values with colored agreement-strength bands in the background. — Side-by-side comparison of the three inter-annotator agreement metrics on the worked example. Cohen's kappa for the A-B pair falls in the moderate range, which reflects the specific divergence in base rates between those two annotators. Fleiss kappa and Krippendorff alpha agree to three decimal places because the data is complete and nominal, placing the three-rater agreement in the moderate range according to Landis-Koch benchmarks. Background bands show the conventional interpretation regions.

Let's demonstrate handling missing data, a key strength of Krippendorff's alpha. We remove two ratings and observe that alpha remains computable and close to the complete-data value.

In[15]:

Code

# Simulate missing data: remove some annotations
annotations_missing = annotations.copy().astype(float)
annotations_missing[2, 1] = np.nan  # Item 3, rater B missing
annotations_missing[7, 2] = np.nan  # Item 8, rater C missing


def krippendorff_alpha_missing(annotations, categories):
    """Alpha with missing data handling."""
    n_items, n_raters = annotations.shape
    k = len(categories)

    coincidence = np.zeros((k, k))

    for i in range(n_items):
        # Filter out NaN values
        valid = annotations[i, ~np.isnan(annotations[i, :])]
        n_valid = len(valid)

        if n_valid < 2:
            continue

        for r1, r2 in itertools.product(range(n_valid), range(n_valid)):
            if r1 != r2:
                c1 = np.where(categories == valid[r1])[0][0]
                c2 = np.where(categories == valid[r2])[0][0]
                coincidence[c1, c2] += 1

    total = np.sum(coincidence)
    D_o = (total - np.trace(coincidence)) / total

    probs = np.sum(coincidence, axis=1) / total
    D_e = 1 - np.sum(probs**2)

    return 1 - (D_o / D_e)


alpha_missing = krippendorff_alpha_missing(annotations_missing, categories)

Out[16]:

Console

Annotations with missing data (NaN):
[[ 2.  2.  2.]
 [ 2.  2.  1.]
 [ 1. nan  1.]
 [ 2.  2.  2.]
 [ 1.  2.  1.]
 [ 0.  0.  0.]
 [ 2.  1.  2.]
 [ 1.  1. nan]
 [ 2.  2.  2.]
 [ 0.  2.  0.]]

Krippendorff's Alpha with missing data: 0.467
Original (complete data): 0.564

The alpha coefficient remains stable even with missing annotations. This makes it suitable for real-world annotation projects where not every rater labels every item. Cohen's kappa and Fleiss' kappa would require you to either drop items with missing data or impute values before computation.

Let's now compute pairwise Cohen's kappa for all three annotator pairs to show how agreement varies across pairs.

In[17]:

Code

# Compute pairwise Cohen's kappa for all annotator pairs
pair_labels = ["A vs B", "A vs C", "B vs C"]
pair_kappas = []
pair_p_o = []
pair_p_e = []

for r1, r2 in [(0, 1), (0, 2), (1, 2)]:
    kappa_val, po, pe = cohens_kappa_manual(
        annotations[:, r1], annotations[:, r2], categories
    )
    pair_kappas.append(kappa_val)
    pair_p_o.append(po)
    pair_p_e.append(pe)

Out[18]:

Console

A vs B: kappa=0.492  (P_o=0.700, P_e=0.410)
A vs C: kappa=0.844  (P_o=0.900, P_e=0.360)
B vs C: kappa=0.355  (P_o=0.600, P_e=0.380)

Out[19]:

Visualization

Bar chart of pairwise Cohen kappa values for annotator pairs A-B, A-C, and B-C with the Fleiss kappa reference line. — Pairwise Cohen's kappa values for all three annotator combinations. The A vs C pair shows the highest agreement among binary comparisons, while A vs B shows the lowest. The variation across pairs illustrates why aggregating over raters with Fleiss kappa or Krippendorff alpha provides a more stable summary when you have more than two annotators.