Exact Match and F1 › Adversarial Examples and Metric Gaming
Language AI HandbookAll chapters Exact Match and F1 › Adversarial Examples and Metric GamingOne section · Sections in this chapter10 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Any evaluation metric that models are trained to optimize will eventually be gamed. Exact Match and F1 are no exception, and understanding how they can be gamed is important for understanding what they measure.
For Exact Match, the most straightforward gaming strategy is to memorize the distribution of reference answer formats in the training set. If training answers always use the short form "US" rather than "United States", a model can learn to always produce "US" and reach high EM on the training distribution, even if it would produce "United States" on a different distribution. This is a form of distributional overfitting that EM cannot detect on its own.
For F1, the gaming strategy is more subtle. Because F1 rewards every correct token regardless of incorrect tokens, a model can increase F1 by generating longer answers that include all the reference tokens plus additional text. In the limit, outputting the entire document as the answer reaches recall = 1.0 for any question, degrading precision but often still achieving moderate F1. Benchmark evaluations that use short reference answers are particularly susceptible to this, as a model that generates three sentences while the reference is one noun phrase will reach zero EM but possibly 30-50% F1 by chance.
Adversarial datasets have been specifically designed to expose these weaknesses. The AddSent adversarial QA dataset adds confusing sentences to context paragraphs that contain answer-like strings. Models that rely on surface-form matching rather than real comprehension sharply degrade on such datasets. Exact Match drops more than F1 on adversarial examples, because models can still capture some correct tokens even when fooled about the right answer span.
These vulnerabilities explain why modern NLP evaluation has moved increasingly toward human evaluation, LLM-as-judge protocols, and multi-metric evaluations that combine EM and F1 with semantic similarity scores. No single metric captures all dimensions of answer quality, and gaming one metric typically degrades performance on complementary metrics. Reporting an ensemble of metrics makes gaming harder and gives a more reliable picture of model capability.