Benchmark Design › Limitations and Practical Considerations

Benchmark design is inherently a compromise between competing desiderata, and no benchmark is perfect. Understanding the limitations helps you use benchmarks responsibly and design future ones better. The limitations are not bugs to be fixed but basic tensions between what we want benchmarks to do and what they can do.

The hardest limitation is the gap between task performance and underlying capability. A model that scores 90% on an NLI benchmark has demonstrated something: it can handle the surface patterns present in NLI data. Whether this reflects a general capacity for logical reasoning that transfers to other domains is a separate empirical question. The history of NLP is full of examples where models seemed to solve a task but were exploiting dataset artifacts: models that "understood" reading comprehension by locating the sentence in the passage containing the most question words, or that performed "sentiment analysis" by looking for opinion words without understanding context. Each time a new benchmark detects this, it reflects the same pattern: the task format created a proxy for the capability rather than measuring the capability directly. The challenge is that proxy traps are often invisible during benchmark construction and only become apparent after careful analysis or adversarial probing.

A second limitation is the difficulty of measuring robustness. A benchmark measures average performance across its examples, but that average can be high while robustness is low. Models that score well on average may still fail dramatically on a small fraction of examples that differ in systematic ways from the training distribution. Adversarial examples, domain shift, and distribution mismatch can cause dramatic performance drops that a standard benchmark would not detect. Robustness-focused benchmarks like AdvGLUE, ANLI, and Winogrande attempt to address this, but they introduce their own biases through the adversarial construction process. A model reliable to the specific adversarial patterns in AdvGLUE may still be fragile to other types of perturbation.

Benchmark saturation is perhaps the most visible limitation. Once the entire research community is aware of a benchmark's existence, there is systematic pressure to overfit to it. Even without deliberate gaming, the publication cycle rewards high benchmark scores; models are selected and tuned based on validation performance; and the same datasets circulate through enough training pipelines that contamination becomes ubiquitous. The result is that a benchmark's discriminative value decays over time in proportion to its popularity. This creates a paradox: the most influential benchmarks are the ones most likely to lose validity, because influence increases optimization pressure.

Finally, benchmarks operationalize the priorities and biases of their creators. English-dominant benchmarks have driven a disproportionate focus on English-language capabilities. Benchmarks based on Wikipedia and academic text favor models trained on formal written language. Benchmarks designed by academic researchers may not reflect the tasks that matter most to practitioners or to the diverse populations who use AI systems. For example, a model that scores highly on academic reasoning benchmarks may perform poorly on code generation, customer service dialogue, or multilingual translation, tasks that matter greatly in practice but are underrepresented in academic evaluation suites. Addressing these biases requires deliberately broadening who creates benchmarks and for what purposes, which is a harder organizational and sociological challenge than any of the technical ones covered in this chapter.