Exact Match and F1 › Normalization for Matching

Raw string matching often proves too brittle for practical evaluation. The same semantic content can be expressed through different surface forms: "United States" versus "US" versus "U.S."; "3:00 PM" versus "15:00"; "cannot" versus "can't". Normalization turns both predictions and references into canonical forms before comparison, reducing false negatives caused by formatting variations rather than semantic errors.

The need for normalization arises from the basic variability of natural language. Human annotators exhibit inconsistencies in capitalization, punctuation usage, and abbreviation conventions. Even within a single dataset, the same answer might appear as "the United States", "The United States", "United States", and "US" across different examples. Without normalization, a model trained on one variant and evaluated on another would appear to fail despite understanding the content perfectly.

Normalization also addresses systematic differences between model outputs and human annotations. Language models trained on large corpora often learn specific formatting conventions that differ from dataset annotation guidelines. A model might consistently include definite articles ("the President") while the dataset omits them ("President"), or vice versa. Normalization prevents these systematic biases from appearing as comprehension failures.

The most widely used normalization pipeline, as standardized in the SQuAD evaluation script, applies four transformations in sequence:

• Case folding: Converting all text to lowercase eliminates case sensitivity. "The White House" and "the white house" become equivalent. However, case sometimes carries semantic information (e.g., "Apple" the company versus "apple" the fruit), so domain-appropriate caution is warranted. In biomedical text, gene names are often case-sensitive: "p53" (the tumor protein) differs from "P53" (which might refer to a different entity or be an error). Case folding should be applied with understanding of the domain's conventions.

• Punctuation removal: Stripping punctuation characters prevents mismatches due to optional periods, commas, or quotation marks. Implementation typically uses regex patterns like [^\w\s] to remove non-alphanumeric characters. This handles variations like "U.S.A." versus "USA" or "don't" versus "dont". However, punctuation can be real in some contexts. In programming languages, "print()" differs semantically from "print", and in mathematics, "x-y" differs from "xy".

• Article removal: In question answering, leading articles ("a", "an", "the") often do not affect answer correctness. "the Eiffel Tower" and "Eiffel Tower" are functionally equivalent in most contexts. Articles can also vary based on sentence position: a phrase appearing at the beginning of a sentence might have "The" capitalized, while the same phrase mid-sentence might not. Removing articles reduces these positional artifacts.

• Whitespace normalization: Collapsing multiple spaces into single spaces and trimming leading/trailing whitespace ensures that "Paris " and "Paris" match. This addresses formatting inconsistencies common in web-extracted text or OCR output, where extra spaces might appear around punctuation or at line breaks.

These four transformations are applied to both the prediction and all references before any comparison. The order matters: applying punctuation removal before whitespace normalization avoids edge cases where removing a period leaves a double space. The specific regex patterns matter too. The SQuAD evaluation script removes characters matching [^a-zA-Z0-9\s] after lowercasing, which preserves digits but removes all other punctuation. A different regex that also removes digits would change scores on numerical answer questions.

Different tasks require specialized normalization rules that reflect the specific conventions of the domain:

• Numerical equivalence: "3.14" should match "3.140" and potentially "3,14" (European decimal notation). Date formats present particularly rich variation: "2024-03-15", "15/03/2024", "March 15, 2024", and "15th March 2024" all stand for the same day. Time formats vary similarly: "3:00 PM", "15:00", and "3pm" may be equivalent depending on context. Implementing reliable numerical normalization requires parsing and canonicalization rather than simple string substitution. Libraries like dateutil and babel give locale-aware parsing that can handle these variations systematically.

• Acronym expansion: "NATO" versus "North Atlantic Treaty Organization" or "FBI" versus "Federal Bureau of Investigation". In some domains, such as legal or medical text, acronyms and their expansions are used interchangeably. Normalization might involve expanding common acronyms or contracting full names to standardized short forms, though this requires domain-specific knowledge bases to avoid errors. A general-purpose acronym expander would need to handle the fact that "PT" can mean physical therapy, part time, point, or the Portuguese abbreviation for Portugal depending on context.

• Unicode normalization: As discussed in Part I, Chapter 2 on Text Normalization, Unicode characters can have multiple equivalent representations. The character "é" can be represented as a single codepoint (U+00E9) or as "e" plus combining acute accent (U+0065 U+0301). Unicode normalization forms (NFC, NFD, NFKC, NFKD) ensure consistent representation before matching. This is important for multilingual evaluation, where different input methods or text sources might use different Unicode representations for the same visual character. NFKC normalization additionally handles compatibility equivalences such as fullwidth characters, which appear in CJK text and would prevent matching their halfwidth equivalents without normalization.

Every normalization rule discards information. While this reduces false negatives (correct answers marked wrong due to formatting), it risks increasing false positives (incorrect answers marked right due to over-normalization). You face a basic trade-off between recall (catching all correct answers) and precision (avoiding incorrect matches).

Consider these specific risks:

• Removing all punctuation makes "Dr." (Doctor) equivalent to "Dr" (Drive in addresses), potentially marking a prediction as correct when it refers to a street rather than a person.
• Case folding makes "US" (United States) equivalent to "us" (pronoun), confusing geographical entities with grammatical words.
• Article removal might equate "a tiger" (any tiger) with "the tiger" (a specific tiger), losing important semantic distinctions of definiteness.
• Aggressive numerical normalization might equate "1.5" (one and a half) with "1,5" (one and a half in European notation) when the latter means "one thousand five" in US notation, or vice versa.

The appropriate level of normalization depends on the task's precision requirements. Medical entity extraction might preserve case and punctuation carefully, as "mg" (milligrams) and "Mg" (magnesium) are critically different, and "5.0" versus "5,0" could stand for dosage errors. General web question answering might aggressively normalize to match human answer variations, accepting that some semantic precision is lost for the sake of robustness.

One underappreciated risk of aggressive normalization is that it can inflate benchmark scores in ways that create false confidence. If a benchmark applies very generous normalization, models can appear to reach near-human performance even when they are still making real errors. This is one reason that benchmark scores have become less reliable indicators of real-world capability over time: as normalization becomes more permissive, the gap between "passes the metric" and "gives a useful answer" widens. The recent trend toward human evaluation and LLM-as-judge evaluation methods is partly a response to this problem.