Benchmark Design › Implementation: Building a Mini Benchmark
Language AI HandbookAll chapters Benchmark Design › Implementation: Building a Mini BenchmarkOne section · Sections in this chapter10 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's make these principles concrete by building a small benchmark from scratch. We will design a simple factual question answering benchmark, implement annotation quality checks, and validate some of its statistical properties.
Setting Up the Dataset
We start by constructing a small annotated dataset. In a real benchmark project, this would come from human annotators; here we will create a structured simulation to illustrate the mechanics.
import random
from dataclasses import dataclass, field
import numpy as np
random.seed(42)
np.random.seed(42)
@dataclass
class QuizQuestion:
question_id: str
question: str
choices: list[str]
correct_index: int
category: str
difficulty: str # easy / medium / hard
annotator_labels: list[int] = field(default_factory=list)
# Build a small dataset of factual questions across three categories
questions_data = [
# Geography - Easy
(
"g1",
"What is the capital of France?",
["Berlin", "Paris", "Madrid", "Rome"],
1,
"geography",
"easy",
),
(
"g2",
"Which ocean is the largest?",
["Atlantic", "Indian", "Arctic", "Pacific"],
3,
"geography",
"easy",
),
(
"g3",
"How many continents are there?",
["5", "6", "7", "8"],
2,
"geography",
"easy",
),
# History - Medium
(
"h1",
"In which year did World War II end?",
["1943", "1944", "1945", "1946"],
2,
"history",
"medium",
),
(
"h2",
"Who was the first president of the United States?",
[
"John Adams",
"Thomas Jefferson",
"George Washington",
"Benjamin Franklin",
],
2,
"history",
"medium",
),
(
"h3",
"The French Revolution began in which decade?",
["1770s", "1780s", "1790s", "1800s"],
1,
"history",
"medium",
),
# Science - Hard
(
"s1",
"What is the atomic number of carbon?",
["4", "6", "8", "12"],
1,
"science",
"hard",
),
(
"s2",
"Which planet has the most moons in our solar system?",
["Jupiter", "Saturn", "Uranus", "Neptune"],
1,
"science",
"hard",
),
(
"s3",
"What is the speed of light in a vacuum (km/s)?",
["~200,000", "~300,000", "~400,000", "~500,000"],
1,
"science",
"hard",
),
]
questions = [QuizQuestion(*d) for d in questions_data]The dataset covers three categories at three difficulty levels. Even at this small scale, you can see the design decisions: each question has exactly four choices, the correct answers are distributed across different positions, and the difficulty tiers have different amounts of domain knowledge required.
Simulating Multi-Annotator Agreement
Real benchmarks require multiple annotators to label each example, both for quality assurance and for measuring IAA. We simulate this by adding plausible annotator errors at different rates per difficulty tier, then computing inter-annotator agreement.
def simulate_annotators(questions, n_annotators=3, error_rates=None):
"""Simulate annotators with different error rates per difficulty."""
if error_rates is None:
error_rates = {"easy": 0.02, "medium": 0.12, "hard": 0.25}
for q in questions:
labels = []
for _ in range(n_annotators):
err_rate = error_rates[q.difficulty]
if random.random() < err_rate:
# Annotator makes an error: picks a random wrong answer
wrong_choices = [
i for i in range(len(q.choices)) if i != q.correct_index
]
labels.append(random.choice(wrong_choices))
else:
labels.append(q.correct_index)
q.annotator_labels = labels
return questions
questions = simulate_annotators(questions, n_annotators=3)The error rates are realistic: well-trained annotators on easy factual questions make errors about 2% of the time, while hard questions might see 25% error rates even from qualified annotators. These are not random errors from carelessness but cases where the question is hard and the annotator's knowledge is incomplete.
Computing Inter-Annotator Agreement
We compute pairwise Cohen's Kappa for each pair of annotators, then aggregate to get an overall agreement score. This follows exactly the formula derived earlier.
from collections import Counter
from itertools import combinations
def cohens_kappa(labels_a, labels_b, n_classes):
"""Compute Cohen's Kappa between two annotators."""
assert len(labels_a) == len(labels_b)
n = len(labels_a)
# Observed agreement
p_o = sum(a == b for a, b in zip(labels_a, labels_b)) / n
# Expected agreement by chance
count_a = Counter(labels_a)
count_b = Counter(labels_b)
p_e = sum(
(count_a.get(k, 0) / n) * (count_b.get(k, 0) / n)
for k in range(n_classes)
)
if p_e == 1.0:
return 1.0
return (p_o - p_e) / (1 - p_e)
def compute_pairwise_kappa(questions, n_annotators=3, n_classes=4):
"""Compute average pairwise Cohen's Kappa."""
annotator_labels = {
i: [q.annotator_labels[i] for q in questions]
for i in range(n_annotators)
}
kappas = []
for i, j in combinations(range(n_annotators), 2):
k = cohens_kappa(annotator_labels[i], annotator_labels[j], n_classes)
kappas.append(k)
return kappas
kappas = compute_pairwise_kappa(questions)
mean_kappa = np.mean(kappas)Pairwise Cohen's Kappa values: Annotators 1 vs 2: kappa = 0.481 Annotators 1 vs 3: kappa = 0.321 Annotators 2 vs 3: kappa = 0.812 Mean kappa: 0.538 Agreement level: Moderate
Agreement in the moderate-to-substantial range (around 0.65-0.80) is typical for well-constructed factual QA tasks. When agreement falls below 0.6, it is a signal to revisit annotation guidelines. When it falls below 0.4, it typically indicates either a basic ambiguity in the task definition or a guideline failure serious enough to require starting over.
We can visualize how annotator error rates vary by difficulty tier, and what the resulting agreement profile looks like across many simulated annotation rounds. The visualization makes concrete what the numbers imply: easy questions are nearly perfectly agreed upon, while hard questions show large spread. This reflects knowledge differences among annotators rather than carelessness.
Checking Label Distribution
Label balance is a basic but important quality check. Heavily imbalanced benchmarks unfairly favor models that learn to predict the majority class. For MCQA specifically, you want to verify that the correct answer is not systematically associated with any particular option position.
from collections import Counter
# Majority vote: the gold label is whatever the majority of annotators chose
def majority_vote(labels):
count = Counter(labels)
return count.most_common(1)[0][0]
gold_labels = [majority_vote(q.annotator_labels) for q in questions]
gold_distribution = Counter(gold_labels)
# Check balance by category
categories = list(set(q.category for q in questions))
category_difficulty = {
cat: [q.difficulty for q in questions if q.category == cat]
for cat in categories
}Gold label distribution (answer position):
Choice 2: ==== (4 questions, 44%)
Choice 3: ==== (4 questions, 44%)
Choice 4: = (1 questions, 11%)
Difficulty distribution by category:
Geography: {'easy': 3}
History: {'medium': 3}
Science: {'hard': 3}A balanced label distribution means models cannot gain an easy advantage by always picking one position. If you see a strong skew toward one choice position, for example 40% of correct answers are always option B, you have a label-position bias that needs to be corrected. This bias is a common annotation artifact: annotators sometimes write the correct answer before the distractors, and certain question writing styles naturally lead to the correct answer landing in the same position. Randomizing choice order before presentation is the standard fix.
Computing Statistical Power
How many examples do you need to reliably detect a 2 percentage point difference in accuracy between two models? The following analysis frames this as a two-proportion z-test, translating the abstract sample size formula into concrete recommendations.
from scipy import stats
def required_sample_size(p1, p2, alpha=0.05, power=0.8):
"""
Compute required sample size per group to detect difference between
proportions p1 and p2 at significance level alpha with given power.
"""
z_alpha = stats.norm.ppf(1 - alpha / 2) # Two-tailed
z_beta = stats.norm.ppf(power)
p_bar = (p1 + p2) / 2
# Standard two-proportion z-test formula
n = (
(
z_alpha * np.sqrt(2 * p_bar * (1 - p_bar))
+ z_beta * np.sqrt(p1 * (1 - p1) + p2 * (1 - p2))
)
** 2
) / (p1 - p2) ** 2
return int(np.ceil(n))
# Analyze sample size requirements for different baseline accuracies and effect sizes
baseline_accuracies = [0.50, 0.65, 0.80]
effect_sizes = [0.01, 0.02, 0.05] # 1pp, 2pp, 5pp differences Baseline Effect N required
----------------------------------
0.50 0.01pp 39,240
0.50 0.02pp 9,806
0.50 0.05pp 1,565
0.65 0.01pp 35,472
0.65 0.02pp 8,806
0.65 0.05pp 1,377
0.80 0.01pp 24,641
0.80 0.02pp 6,039
0.80 0.05pp 906These numbers explain why major benchmarks contain thousands of examples rather than hundreds. Detecting a 2-percentage-point difference between two models requires several thousand examples even in the favorable case where baseline accuracy is 65%. Benchmarks with only 100-200 examples can only reliably detect very large differences, on the order of 10-15 percentage points, which means that many real improvements in model capability are statistically invisible on small benchmarks. This is a systemic problem in early NLP evaluation that led to many overclaims about model improvements that were within the noise floor of the evaluation instrument.
Null Model Baseline Audit
A necessary validation step: ensure that simple null models cannot achieve unreasonably high scores. If any null model substantially exceeds the theoretical chance level, it signals a bias in the benchmark data that needs to be addressed before release.
def evaluate_model(model_fn, questions):
"""Evaluate a model (function from question to predicted choice index)."""
correct = sum(model_fn(q) == q.correct_index for q in questions)
return correct / len(questions)
def random_baseline(q):
"""Pick a random answer."""
return random.randint(0, len(q.choices) - 1)
def always_first(q):
"""Always pick choice A."""
return 0
def always_second(q):
"""Always pick choice B."""
return 1
def longest_answer(q):
"""Pick the longest answer (spurious surface feature)."""
return max(range(len(q.choices)), key=lambda i: len(q.choices[i]))
# Evaluate null models across 1000 random seeds for stability
random_scores = [
evaluate_model(lambda q: random.randint(0, len(q.choices) - 1), questions)
for _ in range(1000)
]Null model performance audit: Theoretical chance (4-choice): 0.250 Random baseline (mean over 1000 seeds): 0.251 ± 0.150 Always-first baseline: 0.000 Always-second baseline: 0.556 Longest-answer baseline: 0.111
If any null model substantially exceeds the theoretical chance level, it signals a bias in the benchmark data. The longest-answer heuristic exploiting answer length is a classic annotation artifact: annotators sometimes make correct answers longer to ensure they are unambiguous, inadvertently creating a spurious surface feature that models can exploit. Always-A or always-B biases indicate that the correct answer was not randomly assigned to positions, a problem that is easy to fix by shuffling option order during dataset construction.
The following chart visualizes the benchmark scoring profile, which makes it easy to see at a glance whether null heuristics cluster near the chance rate or whether some heuristics are surprisingly effective.
