Evaluation Fundamentals › Evaluation Design

A well-designed evaluation study makes defensible claims about model performance. Poorly designed evaluation studies produce numbers that are easy to misinterpret. The design choices that matter most are test set construction, sample size and statistical power, and experimental controls.

The test set is the foundation of any evaluation. Several properties determine its quality.

Distribution match. The test set should reflect the actual distribution of inputs the model will encounter in deployment. A customer service chatbot evaluated on academic question-answering benchmarks may score well but fail in deployment because the evaluation distribution did not match. Constructing a representative test set requires knowledge of the deployment context: who will use the system, what they will ask, and what kinds of failure modes matter most.

Contamination avoidance. Test data must not appear in the model's training data. This sounds obvious but is increasingly difficult to guarantee when training on large web crawls. A model that has memorized benchmark questions and answers from its training data will score artificially high, not because it has learned the underlying capability but because it has learned the specific test instances. This is called benchmark contamination or training data contamination, and it is a growing problem in large language model evaluation.

Contamination can be detected by checking whether benchmark performance significantly exceeds performance on held-out variants of the same tasks, by running membership inference attacks that test whether specific examples are memorized, or by using time-based splitting where test examples were created after the model's training data cutoff.

Diversity and coverage. A test set should cover the range of inputs the model is expected to handle, including edge cases and challenging examples. A test set that is too easy will not differentiate good models from mediocre ones. A test set that is too narrow will only measure one facet of performance.

Label quality. Incorrect or ambiguous labels produce noisy evaluation signals. For tasks with subjective correct answers, inter-annotator agreement should be measured and reported. For tasks with objective correct answers, the labeling process should be carefully reviewed. Poor label quality is a common source of misleading evaluation results.

Label quality issues are often invisible. If a benchmark contains 5% incorrectly labeled examples, a model that achieves 95% accuracy may be correct on every example with a correct label, but the noise in the labels makes it appear to be making errors. Worse, models that perform better than the labeling accuracy cannot be rewarded even if they are correct, because the benchmark treats the (incorrect) label as ground truth. This phenomenon, sometimes called "label ceiling," means that noisy benchmarks systematically underestimate the performance of the best models while masking the true gap between good and mediocre models.

Temporal and demographic coverage. Test sets that are drawn from a narrow time window or from a single demographic can miss systematic failures. A model that handles English-language queries from American users well might fail on queries in other dialects, languages, or cultural contexts. A model evaluated on news articles from 2020 might have learned to perform well on those specific narratives without generalizing to topics that became prominent later. Building representative test sets requires intentional curation to cover diverse populations, time periods, and topic domains, not just random sampling from whatever data is convenient.

A common mistake is drawing strong conclusions from small evaluation samples. If a model achieves 75% accuracy on a 20-example test set versus 72% on a different model, the difference is almost certainly not statistically significant. Even if one model is better, a 20-example sample cannot reliably detect a 3 percentage-point difference.

The sample size required to detect a difference of magnitude $\delta$ with statistical power $1 - \beta$ and significance level $\alpha$ can be approximated for proportion differences using the one-sample proportion formula:

$$n \approx \frac{z_{\alpha/2}^2 \cdot p(1-p)}{\delta^2}$$

where:

• $n$: the required number of test examples per group (i.e., per model being compared)
• $z_{\alpha/2}$: the critical value from the standard normal distribution for a two-tailed test at significance level $\alpha$; for the conventional $\alpha = 0.05$, this equals 1.96
• $p$: the expected proportion of correct answers, approximated as the average of the two models' expected performance
• $1 - p$: the variance-related term reflecting that proportions near 0.5 have the highest uncertainty and require the largest samples
• $\delta$: the minimum effect size you wish to detect, expressed as a proportion (e.g., $\delta = 0.03$ to detect a 3 percentage-point difference)

The formula shows why evaluation is expensive at high precision: shrinking $\delta$ by half requires four times as many examples, because $\delta$ appears squared in the denominator. For example, to detect a 5 percentage-point difference around 70% baseline accuracy at $\alpha = 0.05$, you need approximately

$$n \approx \frac{1.96^2 \times 0.70 \times 0.30}{0.05^2} \approx 323 \text{ examples.}$$

To detect a 1 percentage-point difference under the same conditions, you would need approximately 8,000 examples. Most benchmark evaluations use hundreds to thousands of test examples for this reason, and results should always be reported with confidence intervals or statistical tests.

The figure below shows how required sample size grows as the effect size you want to detect shrinks, for different baseline performance levels. Notice the rapid growth as effect size drops below 3 percentage points: this is the regime where most real-world model comparisons operate, and where small test sets are most misleading.

Statistical power has direct practical consequences. A team that runs a 100-example evaluation and finds no significant difference between two model versions might conclude that the models are equivalent and ship whichever is cheaper to run. But the test may simply have lacked the power to detect a real difference. This is called a Type II error: failing to detect a true effect. The probability of making a Type II error is $1 - \text{power}$, so at 80% power you are accepting a 20% chance of missing a real difference. For high-stakes deployment decisions, 80% power may not be sufficient.

The practical solution is to plan sample sizes prospectively, before running the evaluation. Decide what effect size is meaningful, what significance level you require, and what power you want. Then calculate the required sample size. This forces you to think carefully about the question before you look at data, which is the right order.

Good evaluation design controls for confounds that might explain performance differences apart from the capability being measured. Common confounds include:

Prompt sensitivity. LLMs can respond very differently to slight variations in prompt wording. A model that performs better on one phrasing might perform worse on an equivalent phrasing. Evaluation results should be averaged over multiple prompt templates, or the specific template used should be clearly documented and justified.

Decoding parameters. Temperature, top-p, top-k, and beam search parameters all affect output quality. Models compared using different decoding strategies cannot be fairly compared. Standardize decoding parameters across compared models.

Model size and compute. A larger model will typically outperform a smaller one on most benchmarks. Comparing model A to model B without controlling for parameter count or training compute conflates architectural quality with scale. Fair comparisons require matched scale or explicit reporting of scale differences.

Format compliance. Benchmarks that require specific output formats (multiple choice letter, JSON output, specific code syntax) will systematically underestimate performance if a model fails to comply with the format even when it knows the correct answer. Parsing and format compliance should be handled robustly, and format failures should be analyzed separately.

Position effects. Multiple-choice benchmarks are subject to position bias: models may systematically prefer the first option, the last option, or the longest option regardless of content. This bias should be measured by permuting answer choices and checking whether model performance changes. Any benchmark design that does not account for position effects is producing partially confounded measurements.

Demonstration effects in few-shot evaluation. When evaluation uses few-shot prompts with example demonstrations, the choice of examples can substantially affect performance. Two researchers using different sets of few-shot examples for the same model on the same benchmark can arrive at scores that differ by several percentage points. Best practice is either to use zero-shot evaluation or to average over multiple randomly sampled sets of demonstrations.