Exact Match and F1 › Limitations and Impact

Exact Match and F1 have shaped the development of extractive NLP systems, but their limitations reveal important tensions in how we define "correctness" for language understanding. These metrics reflect a specific philosophical stance: that language understanding can be evaluated through string comparison, and that the goal of NLP systems is to reproduce human-like text rather than to reach semantic goals.

Both metrics are fundamentally lexical: they compare strings, not meanings. This creates a mismatch between metric optimization and human judgment. A model that outputs "The City of Light" when the reference says "Paris" receives zero credit despite being factually correct. Conversely, a model that outputs "Paris, Texas" when asked about French landmarks receives perfect Exact Match if "Paris" appears in the references, even though the answer is wrong in context.

This limitation motivated the development of BERTScore, which uses contextual embeddings to judge semantic similarity rather than surface form overlap. Modern evaluation pipelines often report both F1 (for strict accountability) and BERTScore (for semantic adequacy), particularly for open-domain question answering where paraphrased answers are common.

The lexical bias also affects model development. When you optimize solely for F1, models learn to produce text that matches the lexical patterns of training data rather than focusing on semantic correctness. This can lead to "surface form overfitting," where models memorize specific phrasings from training data rather than learning to generate equivalent paraphrases. The resulting models perform well on held-out data from the same distribution but fail to generalize when question or answer phrasing shifts.

The choice of tokenization materially affects F1 scores in ways that can obscure true model capabilities. Consider the answer "co-operation" versus "cooperation":

• Whitespace tokenization treats these as single distinct tokens (no overlap), yielding zero F1 despite the clear semantic equivalence.
• Character-level tokenization finds substantial overlap, potentially yielding high F1.
• BPE tokenization from Part V might split these differently depending on training data, leading to unpredictable scoring.

This sensitivity means that F1 scores are not comparable across studies using different preprocessing pipelines. A model achieving 85.2 F1 on SQuAD with aggressive punctuation removal might score 82.1 with strict tokenization, even with identical underlying predictions. When comparing results across papers, it is needed to verify the metric name and the specific tokenization and normalization implementation.

The reproducibility problem created by tokenization variation is more serious than it might appear. A 3-point difference in F1 can determine which model a researcher selects for a downstream task, how a paper ranks in a leaderboard, or whether a model is deemed to have surpassed human performance. If that 3-point difference shows tokenization choices rather than real capability differences, the entire comparison is misleading. The community has made progress toward standardized evaluation scripts, but implementation differences persist, particularly in non-English languages where standard English tokenization rules do not apply.

F1 treats predictions as unordered sets of tokens, which suits entity extraction but fails for structured outputs where word order determines meaning. If asked "Who did what to whom?" with reference "Alice gave Bob a book", the prediction "Bob gave Alice a book" reaches high F1 through token overlap (Alice, Bob, gave, book), despite reversing the semantic roles completely. The prediction describes an entirely different event, yet F1 suggests strong performance.

For such structured prediction tasks, specialized metrics like TACRED relation extraction scoring or tree-based similarity measures become necessary. These metrics consider the relational structure between entities, not just their presence in the text. In coreference resolution, temporal ordering tasks, and relation extraction, the word order carries needed semantic information that token-set comparison discards entirely.

The apparent simplicity of Exact Match belies the complexity hidden in normalization pipelines. Different benchmarks apply different normalization rules, making direct comparison difficult. SQuAD 1.1 uses lowercasing and whitespace normalization; SQuAD 2.0 adds punctuation removal; some implementations strip articles while others preserve them. When comparing models across papers, it is needed to verify that metric implementations match, as "F1" is not a single standardized quantity but a family of measures parameterized by tokenization and normalization choices.

This hidden complexity can lead to reproducibility issues. A reported F1 score of 80.0 might stand for very different underlying capabilities depending on whether the implementation used aggressive normalization (making the task easier) or strict matching (which makes it harder). The research community has moved toward standardized evaluation scripts, but variation remains common, particularly in legacy work or domain-specific applications.

Despite these limitations, Exact Match and F1 remain indispensable for development and debugging. Their interpretability allows researchers to inspect specific failure modes: a model consistently achieving 0.0 EM but 0.8 F1 is capturing the right entities but failing on exact boundaries or formatting, suggesting issues with span selection mechanisms rather than comprehension. This diagnostic clarity explains why these metrics persist as standard benchmarks even as more advanced evaluation methods emerge.

One often-overlooked limitation is that both EM and F1 treat the reference answers as ground truth, yet reference quality varies considerably across datasets and annotators. Studies on SQuAD have found inter-annotator agreement on answer spans of around 90%, meaning that roughly 10% of questions have ambiguous or contested gold answers. When you evaluate a model against these contested references, you are not measuring whether the model found the "correct" answer, but whether it found the answer one particular annotator preferred.

This reference quality problem compounds with the multi-answer protocol. Datasets that give multiple references per question acknowledge annotator disagreement, but the number of references provided is often small (SQuAD gives 3-5 answers per question from the development set). If a model finds a sixth equally valid answer, it receives no credit. The reported EM and F1 scores thus reflect model capability and the completeness of the reference set, which varies by question and by dataset construction methodology.