Exact Match and F1 › Token-Level F1

While Exact Match gives a binary correct/incorrect signal, F1 offers a continuous measure of overlap between prediction and reference at the token level. This granularity proves needed when answers can vary in length, when multiple entities appear in a response, or when we want to reward partial correctness.

The transition from Exact Match to F1 shows a shift in evaluation philosophy. Exact Match asks "Is this right?" while F1 asks "How right is this?" This shift acknowledges that in many extraction tasks, getting most of the answer correct has practical value. A system that extracts "Alexander Bell" when the full answer is "Alexander Graham Bell" has identified the correct person and might let useful downstream processing, even if it missed the middle name. Exact Match would discard this progress entirely, while F1 captures it quantitatively.

The practical motivation for F1 becomes especially clear during model development. If you train a model and watch its Exact Match climb from 0.12 to 0.15 over ten epochs, it is difficult to know whether progress is real or noise. But if F1 climbs from 0.45 to 0.62 over those same epochs, you know the model is consistently capturing more of the right information even when it does not yet nail the exact boundaries. F1 acts as a more sensitive signal during training. This gives gradient information about the direction of improvement that binary EM cannot.

F1 derives from the classic information retrieval metrics of precision and recall, adapted to compare token sets rather than document collections. These concepts emerged from library science and early search engines, where the problem was retrieving relevant documents from large collections. In question answering, we adapt them to retrieve relevant tokens from answer strings.

Precision measures what fraction of the predicted tokens were correct, meaning they appeared in the reference:

$$\text{Precision} = \frac{|\text{prediction\_tokens} \cap \text{reference\_tokens}|}{|\text{prediction\_tokens}|}$$

where:

• $\text{prediction\_tokens}$: the set of tokens appearing in the model's predicted answer
• $\text{reference\_tokens}$: the set of tokens appearing in the reference answer
• $\cap$: the set intersection operator, yielding tokens common to both sets
• $|\cdot|$: set cardinality (the count of elements in the set)

Precision answers the question: "Of everything the model said, how much was correct?" High precision indicates the model is conservative, rarely including incorrect information, even if it misses some correct details. In medical extraction, high precision might be important because hallucinated drug names or dosages could lead to dangerous recommendations.

Recall measures what fraction of the reference tokens were successfully predicted:

$$\text{Recall} = \frac{|\text{prediction\_tokens} \cap \text{reference\_tokens}|}{|\text{reference\_tokens}|}$$

where:

• $\text{prediction\_tokens}$: the set of tokens appearing in the model's predicted answer
• $\text{reference\_tokens}$: the set of tokens appearing in the reference answer
• $\cap$: the set intersection operator, yielding tokens common to both sets
• $|\cdot|$: set cardinality (the count of elements in the set)

Recall answers the question: "Of everything that should have been said, how much did the model include?" High recall indicates complete coverage. In legal document analysis, high recall ensures no relevant clauses are missed, even if some irrelevant text is included.

F1 Score is the harmonic mean of precision and recall, giving a single metric that balances both concerns:

$$\text{F1} = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$$

where:

• $\text{Precision}$: the precision of the prediction (fraction of predicted tokens that are correct)
• $\text{Recall}$: the recall of the prediction (fraction of reference tokens that were predicted)
• The harmonic mean balances precision and recall by penalizing cases where one metric is much higher than the other

Why the harmonic mean rather than the arithmetic mean? The arithmetic mean of precision and recall would reward a trivially high-recall system: if the model outputs the entire document as its answer, recall is 1.0 and the arithmetic mean is at least 0.5 regardless of precision. The harmonic mean is always less than or equal to the arithmetic mean, and it approaches zero whenever either component approaches zero. This property means a model cannot "game" F1 by simply outputting everything it sees.

The harmonic mean punishes extreme imbalances more aggressively than an arithmetic mean would. A model that reaches 100% precision by predicting only one correct token (but missing nine others in the reference) receives low F1 due to poor recall, accurately which shows its incomplete performance. Conversely, a model that predicts all reference tokens plus ten irrelevant tokens would have perfect recall but low precision, also resulting in low F1.

This balancing act makes F1 particularly suitable for development and hyperparameter tuning. Unlike Exact Match, which gives no gradient of improvement (a model is either right or wrong), F1 gives partial credit that correlates with human judgments of answer quality. When optimizing a model, seeing F1 increase from 0.4 to 0.6 indicates real progress in balancing comprehensiveness against accuracy, even if Exact Match remains zero.

The calculation of F1 depends critically on how we define a "token." Different tokenization approaches yield different F1 scores for the same string comparison, and this variation changes the substance of the result. It shows basic questions about what constitutes a real unit of meaning in language. Two papers can report the same model achieving "82.3 F1 on SQuAD" and "79.1 F1 on SQuAD" for entirely different reasons: one uses aggressive punctuation stripping, the other does not. The metric name alone does not fully specify what was measured.

The three main tokenization strategies each have distinct behaviors and tradeoffs:

Whitespace tokenization splits on spaces to produce word-level tokens. "New York City" becomes ["New", "York", "City"]. This approach misses multi-word entities and punctuation attachment. It treats "New York" as two separate tokens, potentially matching "York" in isolation if it appears elsewhere in a prediction, even though "York" alone refers to a different entity than "New York". Despite its simplicity, whitespace tokenization is the SQuAD standard because it aligns well with human intuitions about word boundaries and produces results that match what readers expect.

Punctuation-aware tokenization separates punctuation into distinct tokens. "U.S.A." might become ["U", ".", "S", ".", "A", "."] or ["U.S.A.", "."] depending on rules. As we explored in Part I, Chapter 5 on Word Tokenization, these choices materially impact overlap calculations. If periods are separate tokens, then "U.S.A." and "USA" share no tokens, yielding zero F1. If periods are stripped, they might match perfectly. Punctuation-aware tokenization is useful when punctuation carries meaning, as in code or mathematical notation, but introduces complexity when punctuation is stylistic.

Subword tokenization using BPE or WordPiece from Part V splits rare words into smaller units. "hypersensitivity" becomes ["hyper", "##sens", "##itivity"]. F1 calculated on subwords differs substantially from word-level F1. Two answers might share no whole words but share many subwords, or vice versa. This complicates evaluation of models that generate text at the subword level, as the tokenization mismatch between generation and evaluation can distort scores. In practice, evaluation is almost always done at the word level, regardless of what tokenization the model uses internally.

Standard practice for F1 evaluation in question answering uses whitespace tokenization after lowercasing and removing punctuation, striking a balance between linguistic validity and implementation simplicity. However, for tasks involving code, chemical formulas, or specialized domains, custom tokenization rules may be necessary to ensure real token boundaries. In chemical named entity recognition, "H2SO4" should not be split on whitespace or punctuation, as the subscript and element symbols form an indivisible unit.

The choice of tokenization also affects the interpretation of F1 scores. Word-level F1 aligns with human intuition about what constitutes a correct answer, while subword F1 might give credit for morphological similarity that does not reflect semantic correctness. A model predicting "run" when the answer is "running" might score partial credit under subword tokenization (sharing "run"), but these might be distinct answers in a question about verb tense.

When multiple reference answers exist, F1 evaluation typically takes the maximum score across all references:

$$\text{F1}(prediction, references) = \max_{r \in references} \, \text{F1}(prediction, r)$$

where:

• $prediction$: the predicted answer string generated by the model
• $references$: the set of all valid reference answers for the question
• $r$: a single reference answer from the set of references
• $\max_{r \in references}$: the maximum operator selecting the highest F1 score achieved against any single reference

This "best-match" approach ensures that models are not penalized for selecting any valid answer, but it can obscure cases where models consistently pick suboptimal answers (shorter or less complete variants), even when better options exist. If the reference set contains both "John F. Kennedy" and "Kennedy", a model that always predicts just the last name will reach perfect F1 scores despite giving less specific information than available in the text.

The max operation also complicates aggregate statistics. When reporting dataset-level F1, researchers typically macro-average across questions, but the per-question max means that the final score is an upper bound on model performance rather than an average. A model that randomly selects among valid answers will appear as good as one that consistently selects the most complete answer, masking important behavioral differences.

There is a subtle but important question about whether to compute max over the references before or after averaging across questions. The standard approach takes max first (per question) and then averages, which gives the most lenient evaluation. An alternative approach would be to compute F1 against all references, then average, which is stricter. These two approaches can produce substantially different numbers on the same model predictions, yet both would be legitimately called "average F1." When reading papers or comparing results, it is worth confirming which convention was used.