Evaluation Challenges › Limitations and Impact

The challenges described in this chapter are not isolated problems; they are structural features of how evaluation works when the things being evaluated are large, opaque systems trained on massive and poorly documented data. Understanding this is important for calibrating how much confidence to place in published benchmark results.

Benchmark contamination has become a serious problem in the field. As training data scales increase, the probability that any given public benchmark item appears somewhere in the training corpus approaches certainty. This does not mean evaluation is useless, but it does mean that claims of state-of-the-art performance on standard benchmarks come with a built-in asterisk. The responsible interpretation of benchmark scores requires checking what contamination mitigation the team performed, whether the benchmark was held out from training and fine-tuning data, and whether reformulated variants show consistent performance. Without this information, a benchmark score tells you very little about what the model would do on novel instances of the same task.

Evaluation brittleness connects to a deeper question about what language model capabilities amount to. If a model's performance on a task varies by 20 percentage points depending on how the task is phrased, it is reasonable to ask whether the model "has" the capability in any stable sense. From a practical standpoint, brittleness matters enormously for deployment: production systems encounter prompts that differ from the carefully curated prompts used in evaluation, and high benchmark scores achieved under favorable prompt conditions do not guarantee reliable performance in the field. Users do not submit prompts in the canonical evaluation format; they ask questions in their own words with their own formatting conventions. A model that is brittle to surface form variation may deliver inconsistent experiences to different users asking for the same thing.

Metric gaming creates a trust problem for the field as a whole. When benchmark scores become the currency of competition, the incentive to optimize specifically for those benchmarks overwhelms the incentive to develop broadly capable systems. This has happened repeatedly in NLP. The appropriate response is not cynicism about benchmarks but rather methodological sophistication: using benchmarks as one input among many, understanding their limitations, and maintaining healthy skepticism about results that seem too good. Specifically, when a model achieves a large improvement on a benchmark without a corresponding improvement on related tasks or in human evaluations, the most likely explanation is gaming or contamination rather than a capability breakthrough.

The practical guidance in this chapter amounts to a simple principle: treat evaluation as a scientific practice, not a reporting exercise. This means studying whether your measurements are valid before reporting them, checking whether results replicate under different conditions, and distinguishing between what your evaluation measures and what matters for your application. It also means being honest in reporting: noting limitations, uncertainty, and conditions under which results may not hold, even when this reduces the apparent impressiveness of results. The field's ability to make real progress depends on accurate signals about where models stand, not on optimistically interpreted numbers that obscure the gap between current capability and what we need.