Exact Match and F1 › Worked Example

Let's examine how these metrics behave with concrete examples from a question answering scenario. Consider the question "Who invented the telephone?" with reference answers and various model predictions. This question allows us to explore how different types of errors, variations, and completions affect scoring.

Reference answers:

• "Alexander Graham Bell"
• "Bell"

These references stand for different levels of specificity: the full name and the surname. Both are correct, but they set different expectations for what constitutes a complete answer.

Model Prediction A: "Alexander Graham Bell"

• Exact Match: 1.0 (matches first reference exactly)
• F1: 1.0 (perfect token overlap)

This is the ideal case: complete, exact correspondence with the most specific reference. The model has retrieved the precise span that matches the annotator's preferred answer.

Model Prediction B: "alexander graham bell"

• Exact Match: 0.0 (case mismatch)
• F1: 1.0 (after lowercasing, tokens match)
• Normalized EM: 1.0 (after case normalization)

This example shows the brittleness of raw Exact Match and the value of normalization. Without normalization, this prediction appears wrong despite being semantically perfect. The divergence between raw EM and F1 here indicates that token content is correct but formatting differs. This is precisely the scenario normalization is designed to address: the model understood the question correctly but produced output in a different case than the annotator expected.

Model Prediction C: "Bell"

• Exact Match: 1.0 (matches second reference)
• F1: 1.0 (single token matches perfectly)

Here the model gives a less specific but still correct answer. Exact Match gives full credit because "Bell" is in the reference set, but this masks the loss of information (the first and middle names). In some contexts, "Bell" might be ambiguous, referring to Alexander Melville Bell (his father) or other historical Bells, but the metric treats it as fully correct. The multi-reference protocol can thus reward underspecified answers.

Model Prediction D: "Alexander Bell"

• Exact Match: 0.0
• Tokenization: ["Alexander", "Bell"] vs reference ["Alexander", "Graham", "Bell"]
• Precision: $2/2 = 1.0$ (both predicted tokens are correct)
• Recall: $2/3 = 0.667$ (missing "Graham")
• F1: $2 \times (1.0 \times 0.667) / (1.0 + 0.667) = 0.8$

Model Prediction E: "Graham Bell"

• Exact Match: 0.0
• Precision: $2/2 = 1.0$
• Recall: $2/3 = 0.667$
• F1: $0.8$

Despite having different missing tokens, predictions D and E receive identical F1 scores. This shows that F1 captures the quantity of overlap but not the quality or position. Missing the first name versus the middle name might have different semantic significance in some contexts. "Alexander Bell" might be interpreted as a different person than "Graham Bell" by some readers, yet the metric treats these errors as equivalent. Position-insensitive token overlap is a deliberate simplification that makes the metric easy to compute and interpret, but it comes at the cost of ignoring which tokens are missing.

Model Prediction F: "Sir Alexander Graham Bell"

• Exact Match: 0.0
• Precision: $3/4 = 0.75$ ("Sir" is incorrect)
• Recall: $3/3 = 1.0$
• F1: $2 \times (0.75 \times 1.0) / 1.75 = 0.857$

Prediction F reaches higher F1 than D or E (0.857 vs 0.8) despite being factually questionable (the title "Sir" is historically inaccurate for Bell; he received an honorary doctorate but was not knighted). This illustrates how F1 rewards extra correct information even when it accompanies hallucinated content. The model receives partial credit for the extra token "Sir" because F1 only penalizes incorrect tokens in the denominator of precision, not through a separate penalty for fabrication. This property means that F1 alone cannot detect hallucination; it only measures the ratio of correct to incorrect tokens. A model that adds plausible-sounding but wrong information is rewarded rather than penalized as long as it also includes all the correct information.

This hallucination-tolerance is one of the most important practical limitations of F1 as a production evaluation metric. You can build a system that reaches very high F1 by generating verbose answers that happen to include the reference tokens, without those tokens being the actual answer. In production evaluation, you often need to combine F1 with a length penalty or precision threshold to prevent this gaming.