Evaluation Fundamentals › Common Failure Modes in Evaluation

Understanding where evaluation goes wrong is as important as understanding how to design evaluation correctly. Several failure modes recur across language model evaluations, and recognizing them is essential for interpreting published results critically.

Benchmark contamination occurs when questions or answers from a benchmark appear in a model's training data. Because modern language models are trained on enormous web crawls that include academic papers, GitHub repositories, and public datasets, contamination is difficult to avoid entirely. A model that has seen MMLU questions in its training data will score higher on MMLU not because it is more capable, but because it is recalling memorized answers.

Contamination detection is an active area of research. Simple heuristics check for exact or near-exact string matches between benchmark items and training data. More sophisticated methods use perplexity probes: if a model assigns unexpectedly low perplexity to benchmark items relative to semantically similar but uncontaminated text, this is evidence of memorization. Time-based evaluation, where benchmarks are constructed from events or publications after a known training cutoff date, provides the strongest contamination protection but limits the topics that can be evaluated.

The contamination problem is most severe for models that train on the latest snapshot of the web, because benchmark items that were created years ago are well-represented in web crawls. This creates a troubling dynamic: the most capable and widely deployed models are also the most likely to be contaminated on established benchmarks, making it hard to interpret their scores at face value.

Benchmark saturation occurs when models improve so much on a benchmark that performance approaches the ceiling, leaving no room to distinguish between good and excellent models. MMLU reached near-human performance thresholds for frontier models around 2023-2024, meaning that small differences in reported scores no longer carried meaningful signal about capability differences. The benchmark had outlived its usefulness for comparing frontier models, though it remained informative for evaluating smaller or less capable models.

The response to saturation is to develop harder benchmarks and to retire old ones as they saturate. This is an ongoing process: today's challenging benchmark becomes tomorrow's easy one. The community now maintains progressively harder benchmarks for specific capabilities, and some evaluation frameworks explicitly track which benchmarks are "solved" and remove them from aggregate scores to preserve discrimination.

Even without explicit contamination, evaluation can be gamed. Models that are fine-tuned on broad instruction-following datasets that happen to include benchmark-style questions will learn patterns that transfer to benchmark performance without transferring to real-world usefulness. This is sometimes called "benchmark overfitting" and it is structurally similar to the reward hacking problem in RLHF: the model learns to maximize the measured signal without necessarily improving the underlying capability the signal was meant to proxy.

The mitigation is to maintain evaluation sets that are unknown to the training team, to diversify evaluation formats so that no single format can be targeted by training, and to continuously update evaluation sets rather than treating them as fixed.

Metric-goal misalignment occurs when the metric used in evaluation does not reflect the actual goal of the system. A summarization model optimized for ROUGE-L may learn to produce summaries that are lexically similar to reference summaries but miss the key information that a human reader would want. A question-answering model optimized for exact match may fail on questions where the correct answer can be expressed in multiple equivalent forms.

This failure mode is pervasive and difficult to eliminate because it is structural: any metric you can compute is a simplification of the goal. The best defense is to use multiple metrics that make different simplifications, so that a model that games one metric will be caught by another, and to periodically validate automatic metric scores against human judgments to ensure the proxy relationship is holding.