Exact Match and F1 › Metric Selection: When to Use What

Choosing between Exact Match, F1, and their normalized variants requires understanding the task constraints, the cost of false positives versus false negatives, and how model outputs will be consumed downstream. The wrong metric can mislead development efforts or hide necessary failure modes.

The decision between EM and F1 is not arbitrary. It follows from the structure of the task and the consequences of different types of errors:

Use Exact Match when:

• Answers are drawn from closed sets (multiple choice, database IDs), where any deviation is a categorical error.
• Format compliance is necessary (dates must follow ISO 8601, phone numbers must include country codes), and downstream systems will reject malformed inputs.
• Any deviation is a functional error (security codes, medication dosages, financial transaction identifiers), where partial correctness has no value and might be dangerous.
• You want statistically clean comparisons between models, since EM scores have well-understood distributional properties.

Use F1 when:

• Answers can vary in length or composition (list extraction, multi-span answers), and capturing some items is better than capturing none.
• Partial correctness has value (information retrieval, entity extraction), and incomplete information still supports downstream tasks.
• Multiple valid phrasings exist for the same information, and the goal is to measure content coverage rather than format adherence.
• The development signal during training needs to be sensitive enough to detect improvement before the model reaches exact correctness.

Apply normalization when:

• Human annotators show inconsistency in formatting, and you want to measure semantic understanding rather than annotation style alignment.
• The task explicitly allows equivalent forms (dates, measurements, abbreviations), and the evaluation should reflect domain standards of equivalence.
• Model outputs show systematic formatting differences from references (e.g., always including "The" at the beginning of titles), and you want to correct for these biases without retraining.

In practice, most NLP system papers report both EM and F1 alongside their normalized versions. This gives a table that captures performance at multiple levels of strictness. This gives readers the information they need to contextualize scores within their specific use case.

A subtle issue arises when evaluating extractive models that select spans from a provided context. The model might output the correct answer text that appears multiple times in the context but select the wrong instance. For example, if a paragraph mentions "Paris" three times (twice referring to Paris, France and once to Paris, Texas), a model selecting any "Paris" receives full Exact Match credit, even if the surrounding context indicates the wrong entity.

This phenomenon, sometimes called the span ambiguity problem, highlights a limitation of string-based evaluation for extractive tasks. The model's task is to produce the correct text and identify the correct location in the context that supports the answer. If the question asks "What city is the Eiffel Tower in?" and the context mentions Paris, France in the first paragraph and Paris, Texas in the last, selecting the latter "Paris" is clearly wrong, yet Exact Match and F1 would award full points.

More advanced evaluations address this by checking span positions or requiring that the selected span appears in the correct context window. However, standard EM and F1 metrics treat text as isolated strings, divorced from their source context. This limitation means that reported benchmark scores may overestimate true comprehension, as models might exploit statistical correlations between question words and answer locations without real understanding.

The span ambiguity problem has become more relevant as the field has moved from extractive QA (where models highlight a span in the input) to generative QA (where models produce free text). In extractive QA, span positions can be checked. In generative QA, there are no spans at all, and the comparison must be entirely string-based. This shift has made EM and F1 even more approximate as measures of true capability.

When reporting scores across question answering datasets, researchers typically present:

• Exact Match (EM): Percentage of questions with perfect string matches, representing the proportion of fully correct answers.
• F1: Macro-averaged F1 score across all questions, representing the average token overlap quality.
• Normalized variants: EM and F1 after applying standard normalization pipelines, showing performance after accounting for formatting variations.

The gap between EM and F1 indicates answer variability. A large gap suggests many near-misses or acceptable variations, while a small gap indicates that answers are typically short, unambiguous strings where partial credit is rare. For example, in factoid questions with single-word answers like dates or names, EM and F1 will be very close, as there is little room for partial overlap. In multi-sentence explanatory answers, EM might be low while F1 is moderate, showing that models capture key phrases but struggle with complete sentences.

Understanding this gap helps interpret model capabilities. A model with 40% EM and 60% F1 is in a different state than one with 10% EM and 60% F1. The former makes many perfect predictions and some near-misses; the latter rarely gets answers exactly right but often captures relevant content. The appropriate response to these profiles differs: the first model might benefit from better boundary detection, while the second needs basic improvements in answer formation.

Dataset characteristics also determine which metric is most informative. SQuAD 1.1, which draws answers from Wikipedia, produces questions where the reference answers are almost always single noun phrases. On this dataset, the EM/F1 gap is small because short answers leave little room for partial overlap. TriviaQA, which accepts both document-extracted and annotated answers, often shows larger EM/F1 gaps because the reference sets are more diverse. NewsQA, with its longer answers drawn from news articles, shows the largest gaps as models frequently capture the right information in differently bounded spans.