Exact Match and F1 › Relationship to Named Entity Recognition Evaluation
Language AI HandbookAll chapters Exact Match and F1 › Relationship to Named Entity Recognition EvaluationOne section · Sections in this chapter9 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Exact Match and token-level F1 appear in question answering and in the evaluation of Named Entity Recognition (NER) systems, where they take on slightly different interpretations. In NER, the task is to identify spans of text that correspond to entities of specified types (person, organization, location, date, and so on). Evaluation requires token overlap and entity boundaries that match the reference annotation.
Two evaluation protocols are common in NER:
Exact span match requires that the predicted entity span exactly matches the reference span in both position and type. If the reference annotates "General Electric" as an ORGANIZATION and the model predicts "Electric" as an ORGANIZATION, the prediction receives zero credit under exact span evaluation, even though it identified the right entity type and the right sub-span. This strict protocol is used in the CoNLL-2003 shared task evaluation and is in effect Exact Match adapted to the span level.
Token-level F1 computes precision and recall at the level of individual entity tokens, giving partial credit for spans that overlap but do not perfectly align. Under this protocol, predicting "Electric" for "General Electric" would yield recall = 0.5 (one of two tokens found) and precision = 1.0 (the one token predicted is correct), giving F1 = 0.667. Token-level F1 is more forgiving of boundary errors and is often used during development to diagnose whether a model finds the right entity regions even before it gets boundaries precisely right.
The distinction matters because NER errors often fall into predictable patterns. Models frequently find the right entity but predict a slightly shorter or longer span, a phenomenon called "boundary errors." These are qualitatively different from "type errors" (wrong entity type for the right span) or "false positives" (predicting an entity where none exists). By reporting both exact span F1 and token-level F1, you can distinguish these error types and direct engineering effort accordingly.
The NLP community has not fully standardized whether "partial match" in NER means token-level overlap (any shared tokens count) or span-level overlap (the predicted span must start or end at the same position as the reference). The seqeval library, widely used for NER evaluation, implements entity-level F1 where the entire span must match. The SQuAD evaluation script implements token-level F1. These are different metrics even though both are called "F1 in NER."