Exact Match and F1 › Code Implementation
Language AI HandbookAll chapters Exact Match and F1 › Code ImplementationOne section · Sections in this chapter7 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's implement Exact Match and F1 evaluation with various normalization strategies to see these metrics in practice.
import re
import string
from typing import List, Union
def normalize_answer(
text: str,
lowercase: bool = True,
remove_punctuation: bool = True,
remove_articles: bool = True,
fix_whitespace: bool = True,
) -> str:
"""Apply normalization transforms to text."""
if lowercase:
text = text.lower()
if remove_punctuation:
# Remove all punctuation characters
text = text.translate(str.maketrans("", "", string.punctuation))
if remove_articles:
# Remove leading articles
text = re.sub(r"^(a|an|the)\s+", "", text, flags=re.IGNORECASE)
if fix_whitespace:
# Normalize whitespace
text = " ".join(text.split())
return text
def exact_match_score(
prediction: str, references: Union[str, List[str]], normalize: bool = False
) -> float:
"""
Calculate Exact Match score.
Returns 1.0 if prediction matches any reference, 0.0 otherwise.
"""
if isinstance(references, str):
references = [references]
if normalize:
prediction = normalize_answer(prediction)
references = [normalize_answer(ref) for ref in references]
return float(any(prediction == ref for ref in references))
def f1_score(
prediction: str, references: Union[str, List[str]], normalize: bool = False
) -> float:
"""
Calculate token-level F1 score.
Uses whitespace tokenization.
"""
if isinstance(references, str):
references = [references]
if normalize:
prediction = normalize_answer(prediction)
references = [normalize_answer(ref) for ref in references]
# Tokenize by whitespace
pred_tokens = set(prediction.split())
best_f1 = 0.0
for ref in references:
ref_tokens = set(ref.split())
# Calculate overlap
common_tokens = pred_tokens & ref_tokens
if len(common_tokens) == 0:
continue
precision = len(common_tokens) / len(pred_tokens) if pred_tokens else 0
recall = len(common_tokens) / len(ref_tokens) if ref_tokens else 0
if precision + recall > 0:
f1 = 2 * (precision * recall) / (precision + recall)
best_f1 = max(best_f1, f1)
return best_f1The core logic of f1_score is worth reading carefully. The function converts each answer to a set of tokens and computes the intersection. Using a set rather than a list means that duplicate tokens are collapsed: if the prediction contains "Bell Bell", it still counts as one occurrence of "Bell". This set-based approach matches the SQuAD evaluation convention. An alternative multiset approach, which counts token frequencies, would give different results for repetitive answers and is not standard practice for QA evaluation.
Prediction: 'The Eiffel Tower' References: ['Eiffel Tower', 'the Eiffel Tower'] Without normalization: Exact Match: 0.0 F1 Score: 0.800 With normalization: Exact Match: 1.0 F1 Score: 1.000
The normalization process bridges the gap between strict string matching and semantic equivalence. In this example, the article "The" prevents a raw Exact Match, but normalized evaluation recognizes the equivalence. This shows why standard benchmarks like SQuAD apply normalization: without it, models would be penalized for trivial formatting differences that do not reflect comprehension failures.
Notice that raw F1 is already 0.667 in this case, not 0.0. Because "Eiffel" and "Tower" are shared tokens even without normalization, F1 gives partial credit before normalization takes effect. This illustrates the inherent tolerance of F1 compared to EM: even without any normalization pipeline, F1 rewards token-level overlap.
# More complex examples showing partial matching
examples = [
{
"prediction": "Alexander Bell",
"references": ["Alexander Graham Bell"],
"description": "Missing middle name",
},
{
"prediction": "Sir Alexander Graham Bell",
"references": ["Alexander Graham Bell"],
"description": "Extra title (hallucination)",
},
{
"prediction": "Paris, France",
"references": ["Paris"],
"description": "Extra location specifier",
},
{
"prediction": "the French capital city",
"references": ["Paris"],
"description": "Semantic equivalent, no lexical overlap",
},
]Description Prediction EM F1 ------------------------------------------------------------------------------------- Missing middle name Alexander Bell 0.0 0.800 Extra title (hallucination) Sir Alexander Graham Bell 0.0 0.857 Extra location specifier Paris, France 0.0 0.667 Semantic equivalent, no lexical overlap the French capital city 0.0 0.000
Notice how F1 gracefully handles partial overlaps (missing middle names, extra specifiers) while Exact Match remains binary. The scores decrease gradually with the severity of the deviation. This gives a more fine-grained signal than the all-or-nothing Exact Match approach.
The final example shows a necessary limitation: when the prediction is semantically equivalent but lexically distinct ("the French capital city" versus "Paris"), both EM and F1 fail completely, scoring 0.0. This gap motivates the use of BERTScore or semantic similarity metrics for open-ended generation, which we covered earlier in this part. The inability to recognize paraphrases or descriptions is a basic boundary of lexical metrics: they can only judge what is said, not what is meant.
# Implementing SQuAD-style evaluation with multiple references
def squad_evaluate(predictions: dict, references: dict) -> dict:
"""
SQuAD-style evaluation letting multiple valid answers per question.
Args:
predictions: dict mapping question_id -> predicted_answer
references: dict mapping question_id -> list of valid answers
"""
total = len(predictions)
exact_match = 0
f1_total = 0.0
for qid, pred in predictions.items():
refs = references[qid]
if exact_match_score(pred, refs, normalize=True):
exact_match += 1
f1_total += f1_score(pred, refs, normalize=True)
return {
"exact_match": 100.0 * exact_match / total,
"f1": 100.0 * f1_total / total,
}SQuAD-style Evaluation Results: Exact Match: 66.7% F1 Score: 88.9%
This implementation shows the "match-any" protocol used in standard benchmarks. The model receives full credit for "1845" despite three valid variants existing in the reference set, illustrating how benchmark design choices affect reported performance. The flexibility of multiple references accommodates natural variation in how humans express the same information, whether as bare numbers ("1845"), prepositional phrases ("in 1845"), or temporal adverbials ("during 1845").
For "Paris, France" evaluated against the reference "Paris", the normalized evaluation strips punctuation, giving "paris france" versus "paris". The intersection contains "paris" only, so precision is 1/2 = 0.5 and recall is 1/1 = 1.0, yielding F1 = 0.667. This is an informative score: the model found the right city but added an unnecessary qualifier, which is a mild error that deserves partial rather than zero credit.
