Evaluation Challenges › Evaluation Best Practices

Given these challenges, how should you approach evaluation? There is no perfect solution, but there are practices that substantially improve evaluation quality. The key insight is that evaluation is a design problem with tradeoffs, not a lookup problem with known answers. Good evaluation requires understanding what you want to measure, choosing instruments that are valid for that purpose, and being transparent about the gap between your instruments and your actual goals.

No single benchmark captures every aspect of capability. A sound evaluation uses multiple complementary measures:

• Capability-specific benchmarks: Cover different skills (reasoning, knowledge, language understanding, generation quality) separately. Aggregate scores obscure important variation. A model might perform well on factual recall and poorly on multi-step reasoning; an average score hides this distinction.
• Prompt robustness tests: Evaluate performance on multiple formulations of each task. If scores vary widely across formulations, the measurement is unreliable. Wide variance is itself an important finding that should be reported.
• Held-out test sets: Maintain evaluation items that were not used in any model selection decision. Items used for hyperparameter tuning are not held out in any meaningful sense. The distinction between validation and test data is not just terminology; it reflects whether the evaluation provides new information about generalization.
• Adversarial items: Include examples specifically designed to test failure modes, edge cases, and known weaknesses. Benchmarks that test only typical cases miss important reliability properties. Adversarial evaluation is particularly important for safety-critical applications where failures are costly.
• Domain-specific subsets: If your application involves a specific domain (legal, medical, technical), evaluate explicitly on that domain rather than relying on general benchmarks to be representative. General-purpose evaluation results may not transfer.

Rather than treating prompt format as a fixed choice, treat it as an experimental variable. Evaluate each model across a distribution of prompt formats and report the mean and variance. A model that achieves 80% with low variance is more reliable than a model that achieves 85% average with high variance. Variance across prompts is a first-class measurement, not a nuisance to be suppressed.

For comparing models, pair-wise comparisons on identical prompts are more informative than comparing numbers across papers that used different prompts. When reviewing published results, check whether evaluations used matched prompts across models. Two models evaluated with different prompts are not directly comparable, even if the prompts appear similar. Even subtle differences in phrasing, formatting, or few-shot examples can shift results substantially enough to change which model appears better.

A practical workflow is to define a canonical evaluation prompt format, run all models on that format, and then run a random sample of prompt variations to estimate robustness. Reporting both the canonical score and the robustness estimate gives readers far more information than a single number.

For tasks with non-binary outputs, no single metric captures all relevant dimensions of quality. Consider:

• Correctness and fluency are different dimensions for generation tasks; measuring one does not measure the other. A factually wrong but fluently written response scores high on fluency metrics and may score high on automated similarity metrics if it uses the right vocabulary, even though it is useless or harmful.
• Calibration (whether the model's expressed confidence matches its actual accuracy) is important for practical use but is rarely reported in benchmark results. A model that achieves 85% accuracy while being correctly calibrated is more useful than a model that achieves 87% accuracy while being poorly calibrated and overconfident in its errors.
• Consistency across semantically equivalent inputs reveals whether high accuracy reflects understanding or fragile pattern matching. Testing whether a model answers "What is the capital of France?" and "France's capital city is?" identically provides useful information about robustness.
• Error analysis on failed items often reveals more than aggregate accuracy scores; categorizing failure modes by type gives actionable diagnostic information. Knowing that a model fails primarily on questions requiring multi-step arithmetic, rather than failing uniformly, is actionable in a way that a single accuracy number is not.
• Behavioral metrics: For production systems, metrics like response latency, refusal rate, and output length distribution matter alongside quality measures. These are not substitutes for quality metrics but are often just as important for users.

The strict separation between data used for development decisions and data used for final evaluation is a foundational principle that is frequently violated in practice. Any benchmark item that influenced a training decision, a model selection decision, or a hyperparameter choice is no longer an independent evaluation item.

Concretely, this means:

• Never tune on your test set: If you use benchmark performance to choose between model checkpoints, that benchmark has become a validation set, not a test set. The distinction is not just semantic: it changes what the benchmark score tells you. A test-set score estimates generalization performance; a score on data you tuned toward estimates how well you tuned.
• Version your evaluations: When you run a model on a benchmark, record the exact version of the benchmark, the exact prompt format, and the exact evaluation code. Subtle changes to any of these can shift results substantially. A version-controlled evaluation setup ensures that you can reproduce results and that comparisons across time are valid.
• Report all evaluation results: Selective reporting of favorable results is a form of implicit gaming. If you ran 20 evaluation configurations and report only the highest-scoring one, your reported score reflects cherry-picking rather than expected performance. Pre-registering your evaluation plan before running experiments is the most effective guard against this.
• Document what you tuned against: Even if your final evaluation is clean, documenting which benchmarks were used during development helps readers understand what the final evaluation scores represent. This transparency is increasingly expected in high-quality empirical research.

Benchmark scores are estimated from finite samples and carry uncertainty that is rarely reported. A model's reported accuracy of 72.3% on a 1,000-item benchmark is an estimate with a confidence interval. Two models that score 72.3% and 73.1% are not necessarily different; the difference may be within sampling noise.

For a binary outcome (correct or incorrect) with $n$ items and observed accuracy $\hat{p}$, the standard error of the estimate is:

$$\text{SE} = \sqrt{\frac{\hat{p}(1 - \hat{p})}{n}}$$

For $n = 1000$ and $\hat{p} = 0.72$, this gives:

$$\text{SE} = \sqrt{\frac{0.72 \times 0.28}{1000}} \approx 0.014$$

A 95% confidence interval for the true accuracy spans roughly $0.72 \pm 0.028$. Two models with scores of 72.3% and 73.1% are well within this interval of each other and are statistically indistinguishable on this benchmark. Yet papers frequently report such differences as meaningful improvements.

Statistical significance testing, confidence intervals, and bootstrap resampling are standard statistical tools that are underused in NLP evaluation. When comparing models, report whether observed differences are statistically significant. When sample sizes are small (common for expensive-to-evaluate benchmarks), report the uncertainty explicitly. The practice of reporting benchmark scores to one or two decimal places without confidence intervals creates a false impression of precision that the data do not support.

Human performance provides a useful reference point, but requires careful interpretation. Human baselines should be collected under comparable conditions: same instructions, same time constraints, same interfaces. Comparing model performance to human performance under adversarial conditions (humans given confusing instructions, rushed time limits) is misleading. Comparing it to human performance under highly favorable conditions (domain experts with unlimited time) is also misleading in the opposite direction.

Human performance also varies substantially across humans. Reporting aggregate human accuracy masks variation. A more useful comparison is model performance versus human performance at the same demographic and expertise level as the intended use case. A medical question-answering benchmark evaluated against first-year medical students is measuring something different from the same benchmark evaluated against experienced physicians. If you want to know whether a model can help first-year medical students, the first comparison is more informative.

There is also a deeper question about what human performance comparisons mean for language models. Humans and language models are cognitively very different. A human who scores 80% on a benchmark has a model of the world that supports generalization; they can apply their knowledge in new contexts in ways that a benchmark score does not capture. A language model that scores 80% may have achieved that score through very different mechanisms, some of which generalize and some of which do not. The fact that a model matches human performance on a specific benchmark is informative, but it does not mean the model's capabilities are human-like in the ways that matter most for real-world use.

A benchmark is valid if scores on the benchmark are related in a useful way to the capability the benchmark claims to measure. Validity is distinct from reliability (whether the benchmark produces consistent scores under identical conditions). A benchmark can be very reliable (consistent scores) and completely invalid (not measuring the claimed skill at all).

Assessing validity requires answering: What capability does this benchmark claim to measure? What is the evidence that benchmark scores reflect that capability rather than something else (text memorization, prompt-format familiarity, spurious correlations in the dataset)? This question is harder to answer than it sounds, and the answer changes as models and evaluation practices evolve.

The validity of a benchmark decreases as contamination accumulates, as the benchmark format becomes familiar through training, and as researchers optimize systems for the benchmark specifically. A benchmark that was valid when first introduced, because it was testing novel capabilities, may become invalid over time as the field collectively learns how to exploit its structure.

Thinking carefully about validity also means resisting the temptation to equate benchmark performance with real-world utility. A model that achieves high scores on a reading comprehension benchmark may still struggle with the reading comprehension tasks your application requires if those tasks differ from the benchmark distribution. Benchmark performance is evidence about capability, but it is not a substitute for evaluation on your actual task.