Benchmark Design › Step 2: Dataset Collection

With a task definition in hand, you need concrete examples. Dataset collection is labor-intensive and strongly affects on benchmark quality. The choices you make about where examples come from, how they are filtered, and what the size distribution looks like shape every subsequent analysis.

The main collection strategies differ in cost, quality, and potential biases:

Curating from existing sources. This means finding naturally occurring text that instantiates your task. For question answering, this might mean extracting questions from trivia websites, FAQ pages, or Wikipedia talk pages. For reading comprehension, it might mean finding news articles paired with comprehension exercises. Natural Questions, TriviaQA, and WebQuestions all use this approach. Curation is efficient but may introduce selection bias: naturally occurring data reflects the biases of whoever created it, including demographic and cultural assumptions that are invisible to the benchmark creator but systematic in the data.

Crowd-sourcing via human annotation. This means hiring workers via platforms like Amazon Mechanical Turk, Prolific, or Scale AI to create or label examples. Crowd-sourcing gives you control over the task format and allows collection at scale, but quality is uneven and instructions must be extraordinarily clear to get consistent results. Worker demographics also affect data characteristics: crowd workers are not a random sample of language users, and their writing patterns, cultural references, and reasoning styles differ from specialists and from each other.

Expert annotation. For specialized domains like law, medicine, or mathematics, crowd-sourced workers often lack the background to create or label examples correctly. Expert annotation is expensive, sometimes requiring hundreds of dollars per hour for domain specialists, but necessary when the task requires domain knowledge. Benchmarks like PubMedQA and LegalBench rely on expert contributors. Expert annotation typically yields higher-quality labels but smaller datasets, which creates the tradeoff between label quality and statistical power discussed later.

Synthetic generation. Modern LLMs can generate plausible-looking task instances at scale and low cost. Synthetic data is attractive when expert annotation is too expensive or when you need specific edge cases that rarely occur naturally. But synthetic generation carries serious risks. The most basic is circular validity: if you use the same or similar models to generate the benchmark and to evaluate on it, high performance may reflect the evaluator's familiarity with the generator's style rather than broad capability. Synthetic generation is most defensible when the generated examples are subsequently verified and filtered by humans who are unaware of which models will be evaluated on the benchmark.

Hybrid approaches. Most production benchmarks combine strategies. A common pattern is to use an LLM to generate initial candidate examples, then have human experts filter and correct them. This combines the scalability of synthesis with the quality control of human judgment. Another hybrid is to start with naturally occurring text and then have annotators add structure (labels, answer annotations, difficulty ratings) that the original source does not contain. SQuAD follows this pattern: naturally occurring Wikipedia passages were provided to crowd workers who wrote question-answer pairs grounded in each passage.

How you sample examples matters as much as where they come from. A benchmark that overrepresents common cases will favor models that are good at typical examples but may miss systematic failures at the margins. A benchmark that only includes edge cases will be so hard that most models cluster near chance, losing discriminative power for the large middle of the capability distribution.

A well-calibrated benchmark spans the full difficulty distribution. Some easy examples provide a floor, confirming that a model has basic competence and distinguishing it from complete failure. Many medium examples differentiate most models from each other. This provides the gradient signal that research needs to make incremental progress. Some hard examples distinguish elite performance from near-elite, which is especially important as the field matures and many models converge on high average performance.

For topic-based benchmarks like MMLU, which covers 57 subjects ranging from high school biology to abstract algebra to professional law, you must also decide how to weight different topics. Equal weighting treats all topics as equally important; frequency weighting gives more weight to common topics; capability weighting emphasizes tasks where differentiation is most informative. These choices should be made explicitly rather than falling out of convenience sampling. A benchmark whose final score is dominated by a few easy, common topics may rank models differently than one that weights harder or rarer topics equally.

Coverage is another dimension. For language benchmarks, coverage includes language varieties (British vs. American English), genres (formal vs. informal), time periods (current events vs. historical knowledge), and demographic perspectives. Narrow coverage creates a benchmark that measures capability for a specific subpopulation; broad coverage is harder to achieve but produces scores that generalize more widely.

How many examples do you need? The answer depends on the metric precision you need, the variance of model behavior across examples, and the number of sub-categories you want to analyze separately.

For a benchmark with a single aggregate accuracy metric, the confidence interval around a proportion $\hat{p}$ with $n$ examples is approximately:

$$\text{CI} = \hat{p} \pm z \sqrt{\frac{\hat{p}(1 - \hat{p})}{n}}$$

where:

• $\hat{p}$: the observed proportion of correct answers, our estimate of the model's true accuracy
• $n$: the number of benchmark examples
• $z$: the necessary value for the desired confidence level (1.96 for 95%, 1.645 for 90%)

With $n = 100$ examples and $\hat{p} = 0.70$, the 95% confidence interval is roughly $\pm 9$ percentage points, spanning a range from 61% to 79%. This means you cannot reliably distinguish two models whose scores differ by less than 9 points. With $n = 1000$ examples, the interval narrows to $\pm 2.8$ percentage points. With $n = 10000$ examples, it narrows further to $\pm 0.9$ percentage points. Most modern benchmarks use at least 1,000-10,000 examples per task to detect differences of 1-2 percentage points between models.

If you want to report per-category scores and expect meaningful differences within categories, each category needs enough examples to support separate analysis. A rough rule of thumb is 200+ examples per category for statistical reliability at the 5 percentage point level. With only 50 examples per category, even a 10-point difference between categories would not be statistically reliable.

The following chart makes this concrete, showing how the required sample size grows as the detectable effect size shrinks. Notice how detecting a 1-percentage-point difference requires over 10,000 examples at typical accuracy levels.