Evaluation Challenges › Gaming Metrics

When evaluation metrics become targets, they stop being good measures. This insight, often called Goodhart's Law in economics and social science, applies with particular force to machine learning benchmarks. Once a benchmark becomes the standard measure of some capability, the pressure to score well on that benchmark incentivizes behaviors that improve the score without improving the capability.

The mechanism behind Goodhart's Law is straightforward in machine learning terms. A metric is designed to correlate with a capability under the assumption that the metric's designers did not anticipate every possible way to optimize it. Once researchers start actively optimizing for the metric, they explore strategies the designers did not foresee. Some of these strategies improve capability. Others exploit idiosyncrasies of the metric: they raise the measured score without raising (and sometimes while lowering) the underlying capability. Over time, as more effort goes into metric optimization, the fraction of score improvement driven by underlying capability improvement shrinks.

Gaming can be unintentional or deliberate, and it occurs at multiple levels.

Hyperparameter tuning to benchmarks: The standard practice of selecting model checkpoints and hyperparameters based on validation performance can inadvertently select for benchmark-specific behaviors. If the same benchmarks are used for both hyperparameter tuning and final evaluation, the evaluation is no longer measuring generalization; it is measuring how well the configuration was tuned to those specific items. This is a subtle form of data leakage that is easy to commit accidentally. Every time a researcher runs an experiment, observes benchmark performance, and adjusts a hyperparameter accordingly, that benchmark is being used as a target.

Data selection and filtering: Training data curation decisions can implicitly target benchmark performance. If a researcher observes that certain types of training data improve benchmark scores, they may increase the proportion of that data. Over many iterations, training data distributions can shift to favor benchmark-correlated patterns. This process can occur without any intention to game: researchers reasonably prefer training data that improves their benchmarks. The cumulative effect is still that the model is being fit to the benchmark rather than to the underlying task.

Post-processing optimization: Some teams optimize post-processing pipelines (sampling temperatures, output filtering, answer extraction heuristics) specifically for benchmark evaluation. The resulting system may score well but behave very differently in practice where these optimizations are absent. A model evaluated with a temperature of 0 and a specific answer-extraction regex may perform quite differently when deployed with standard generation parameters.

Deliberate training on benchmark-adjacent data: More aggressively, models can be trained explicitly on data that is similar to benchmark items without including the items themselves. This is technically not contamination (the actual test items were not seen), but it optimizes for the evaluation target rather than the underlying skill. Creating training data by paraphrasing benchmark items, or by scraping every resource that covers benchmark-adjacent topics in depth, falls into this category. The distinction between "curating high-quality training data" and "gaming the benchmark" can become uncomfortably blurry.

Model selection from a large candidate pool: If many model variants are trained and only the best-performing one is reported, the reported performance reflects the upper tail of a distribution rather than the expected performance of the training procedure. This is a form of cherry-picking that is especially common when training is expensive and only a few runs can be reported. The solution is to pre-register evaluation plans before training, or to report results across all runs rather than just the best.

As models improve on a benchmark, the benchmark becomes less informative. Once multiple models achieve above 90% accuracy on a task, differences in the remaining items may reflect idiosyncrasies of those items rather than systematic capability differences. This is evaluation saturation.

The response is usually to create harder benchmarks. But this just moves the target: the new benchmark becomes the standard, pressure builds on it, and the cycle repeats. The field has seen this pattern with GLUE, then SuperGLUE, then BIG-Bench, with frontier model performance on each catching up faster than initially expected.

Saturation also creates a false sense of progress. When a benchmark saturates near 100%, observers may conclude that the underlying capability is solved. But the benchmark may have been an imperfect proxy for the capability in the first place, and saturation on the proxy does not mean saturation of the capability itself. Reading comprehension benchmarks, for example, have been "solved" multiple times, yet language models continue to fail on reading comprehension tasks that require common sense, temporal reasoning, or tracking entity states across long documents. The benchmark measured a slice of the capability, and saturating that slice told us only that models had mastered that particular slice.

There is also an underappreciated compositional effect: as more and more research groups evaluate on the same benchmark, the community accumulates detailed knowledge about which prompt formats, which training data characteristics, and which model architectures perform best on that benchmark specifically. This community-level optimization creates a form of collective gaming even when no individual team is acting improperly.

The canonical example in NLP was BLEU score for machine translation. BLEU measures n-gram overlap between a generated translation and reference translations. It was an excellent diagnostic tool when first introduced. But as BLEU became the standard evaluation metric, systems began to be tuned specifically for BLEU performance, and researchers noticed that high-BLEU systems often produced translations that human evaluators found worse than lower-BLEU alternatives. Systems learned to game BLEU by overusing frequent n-grams and avoiding rare words, even when the rare word was the correct translation. A translation that used common synonyms for every term would score well despite being wooden and sometimes inaccurate.

The same dynamic has played out with:

• ROUGE scores for summarization, where systems produce summaries that match reference n-grams but may lack coherence or factual accuracy. A summary that copies the most frequent phrases from the source document will score well on ROUGE regardless of whether it makes sense as a standalone text.
• Accuracy metrics for classification, where high accuracy can be achieved by exploiting class imbalances or spurious correlations. A classifier that always predicts the majority class achieves high accuracy on an imbalanced dataset while being useless for the minority class.
• Perplexity for language modeling, where very low perplexity can result from models that are extremely good at predicting training-data-like text but struggle on out-of-distribution inputs. Optimizing perplexity on a specific domain can degrade performance on other domains even as the single reported number improves.
• Human evaluation scores obtained through crowdsourcing, where annotators can be gamed by systems that produce confident, authoritative-sounding text. Annotators who are not domain experts tend to rate fluent text as accurate even when it is not, and systems optimized for human preference scores in this setting may learn to sound confident rather than to be correct.

Each of these cases follows the same pattern: a metric captures some real aspect of quality, becomes the optimization target, and then decouples from quality as optimization pressure reveals the gap between the metric and the underlying property.