Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Named Entity Recognition › NER Evaluation

Michael BrenndoerferUpdated January 30, 20269 min read

Language AI HandbookAll chapters

Part 6 of 67Sequence Labeling

Chapter 37Named Entity Recognition › NER Evaluation

Reading section 7 of 17 · NER Evaluation · 9 minSection 7 of 17 · 9 min

Section 7 of 17 · 9 min read

Named Entity Recognition › NER EvaluationOne section · Sections in this chapter7 / 17 ↓

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Measuring NER system performance requires care because there are multiple ways to define what counts as a correct prediction. Two distinct evaluation philosophies produce very different numbers for the same system.

Exact Match vs. Partial MatchLink Copied

In exact match evaluation (also called strict evaluation), a prediction is correct only if both the span boundaries and the entity type match the gold annotation exactly. A system that predicts "Tim Cook" as ORG instead of PER gets zero credit. A system that predicts "Cook" instead of "Tim Cook" also gets zero credit, even though it partially overlapped with the correct span.

Partial match evaluation (also called lenient evaluation) relaxes this requirement. It counts a prediction as correct if it overlaps with a gold span, even if the boundaries are imprecise or the type is wrong. Different variants of partial matching include:

Type match: The entity type is correct, boundary is ignored.
Partial boundary match: The predicted span overlaps with the gold span, regardless of type.
Exact type + partial boundary: Type is correct and spans overlap.

Strict vs. Lenient Evaluation

Strict (exact match) evaluation is the standard for NER benchmarks like CoNLL 2003. It requires both the exact span boundaries and the correct type label. Lenient evaluation is used when approximate extraction is acceptable, for example in information retrieval applications where finding the entity at all matters more than exact delineation.

Precision, Recall, and F1Link Copied

NER evaluation uses precision, recall, and their harmonic mean F1, computed over entity spans rather than individual tokens.

Given a set of predicted entity spans $\hat{E}$ and gold spans $E^*$ , the metrics are:

\text{Precision} = \frac{|\hat{E} \cap E^*|}{|\hat{E}|}

\text{Recall} = \frac{|\hat{E} \cap E^*|}{|E^*|}

F_1 = \frac{2 \cdot \text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}

where:

$\hat{E}$ : the set of (span, type) pairs predicted by the system
$E^*$ : the set of (span, type) pairs in the gold annotation
$|\hat{E} \cap E^*|$ : the number of predicted spans that exactly match a gold span (in both boundary and type)

Precision measures how often the system's predictions are correct. Recall measures how often gold entities are found. F1 balances the two. A system that predicts fewer but more accurate entities will have high precision and low recall. A system that aggressively predicts many entities will have high recall and low precision.

NER systems typically report span-level F1 for each entity type separately (per-type F1) as well as a macro-averaged F1 across types and a micro-averaged F1 weighted by entity count.

Let's implement these metrics from scratch:

In[19]:

Code

def ner_metrics(gold_spans, pred_spans):
    """
    Compute precision, recall, and F1 for NER.

    gold_spans, pred_spans: lists of (start, end, label) tuples
    """
    gold_set = set(gold_spans)
    pred_set = set(pred_spans)

    true_positives = len(gold_set & pred_set)
    false_positives = len(pred_set - gold_set)
    false_negatives = len(gold_set - pred_set)

    precision = (
        true_positives / (true_positives + false_positives) if pred_set else 0.0
    )
    recall = (
        true_positives / (true_positives + false_negatives) if gold_set else 0.0
    )
    f1 = (
        2 * precision * recall / (precision + recall)
        if (precision + recall) > 0
        else 0.0
    )

    return {
        "precision": precision,
        "recall": recall,
        "f1": f1,
        "tp": true_positives,
        "fp": false_positives,
        "fn": false_negatives,
    }


# Gold annotations (correct answer)
gold = [
    (0, 2, "PER"),  # "Tim Cook"
    (3, 4, "ORG"),  # "Apple"
    (6, 7, "LOC"),  # "Paris"
    (9, 11, "DATE"),  # "next Tuesday"
]

# System A: good at persons and locations, misses date, wrong type on org
pred_a = [
    (0, 2, "PER"),  # correct
    (3, 4, "PER"),  # wrong type: predicted PER instead of ORG
    (6, 7, "LOC"),  # correct
    # date missed entirely
]

# System B: slightly off boundaries but correct types
pred_b = [
    (0, 2, "PER"),  # correct
    (3, 4, "ORG"),  # correct
    (6, 7, "LOC"),  # correct
    (9, 10, "DATE"),  # partial: "next" only, not "next Tuesday"
]

metrics_a = ner_metrics(gold, pred_a)
metrics_b = ner_metrics(gold, pred_b)

Out[20]:

Console

Gold spans:
  (0, 2, 'PER')
  (3, 4, 'ORG')
  (6, 7, 'LOC')
  (9, 11, 'DATE')

System A predictions (type error on ORG, missed DATE):
  (0, 2, 'PER')
  (3, 4, 'PER')
  (6, 7, 'LOC')
  Precision=0.67  Recall=0.50  F1=0.57
  TP=2  FP=1  FN=2

System B predictions (correct types, one boundary error):
  (0, 2, 'PER')
  (3, 4, 'ORG')
  (6, 7, 'LOC')
  (9, 10, 'DATE')
  Precision=0.75  Recall=0.75  F1=0.75
  TP=3  FP=1  FN=1

System A gets 2 true positives (the PER and LOC spans that exactly match gold), 1 false positive (the wrong-type ORG prediction), and 2 false negatives (the undetected DATE and the wrong-type ORG). System B also gets 2 true positives (PER and LOC), 1 false positive (the partial DATE span), and 2 false negatives (the exact DATE and the missing ORG). Both systems achieve an F1 of 0.57. This shows that type errors and boundary errors have symmetric penalties under exact match scoring.

This strictness is both a strength and a limitation of exact match evaluation. It creates an unambiguous ranking of systems, but it can understate the practical utility of a system that consistently finds entities with slightly off boundaries.

Visualization of Evaluation ErrorsLink Copied

Out[21]:

Visualization

Bar chart showing four error categories in NER evaluation: correct, boundary error, type error, and missed entity. — Four categories of NER evaluation errors under exact match scoring. Correct predictions require both exact span boundaries and the correct entity type. Any deviation, whether a boundary shift, type mismatch, spurious prediction, or missed entity, is counted as an error. Boundary errors and type errors together account for roughly 19% of all entity-level outcomes in this simulation.

Boundary errors and type errors each account for roughly 10% of predictions in this simulation. Boundary errors are the more common failure mode in practice because they often arise from consistent annotation disagreements (whether to include titles, whether to extend a span to include possessives) rather than fundamental model failures.

Comparing Evaluation SchemesLink Copied

Let's directly compare exact match and partial match evaluation on the same predictions to see how much the choice of evaluation scheme affects reported performance:

In[22]:

Code

# Simulated gold annotations for a small test set
gold_annotations = [
    # Sentence 1: "Apple CEO Tim Cook visited Berlin."
    [(0, 1, "ORG"), (2, 4, "PER"), (5, 6, "GPE")],
    # Sentence 2: "Google and Amazon invested $500 million in AI."
    [(0, 1, "ORG"), (2, 3, "ORG"), (4, 6, "MONEY")],
    # Sentence 3: "The WHO published new guidelines in January 2025."
    [(1, 2, "ORG"), (5, 7, "DATE")],
]

# Simulated system predictions (with some errors)
pred_annotations = [
    # Sentence 1: correct on PER and GPE, misses ORG
    [(2, 4, "PER"), (5, 6, "GPE")],
    # Sentence 2: correct on ORG spans, boundary error on MONEY
    [(0, 1, "ORG"), (2, 3, "ORG"), (4, 5, "MONEY")],
    # Sentence 3: correct on both
    [(1, 2, "ORG"), (5, 7, "DATE")],
]

# Aggregate across sentences
all_gold = [span for sent in gold_annotations for span in sent]
all_pred = [span for sent in pred_annotations for span in sent]

overall = ner_metrics(all_gold, all_pred)


def partial_match_metrics(gold_spans, pred_spans):
    """
    Lenient evaluation: a prediction is correct if spans overlap,
    regardless of exact boundaries (type must still match).
    """
    matched_gold = set()
    true_positives = 0

    for pg_start, pg_end, pg_label in pred_spans:
        for i, (gg_start, gg_end, gg_label) in enumerate(gold_spans):
            if (
                pg_label == gg_label
                and pg_start < gg_end
                and pg_end > gg_start
                and i not in matched_gold
            ):
                true_positives += 1
                matched_gold.add(i)
                break

    false_positives = len(pred_spans) - true_positives
    false_negatives = len(gold_spans) - true_positives

    precision = true_positives / len(pred_spans) if pred_spans else 0.0
    recall = true_positives / len(gold_spans) if gold_spans else 0.0
    f1 = (
        2 * precision * recall / (precision + recall)
        if (precision + recall) > 0
        else 0.0
    )
    return {"precision": precision, "recall": recall, "f1": f1}


exact = ner_metrics(all_gold, all_pred)
partial = partial_match_metrics(all_gold, all_pred)

Out[23]:

Visualization

Grouped bar chart comparing precision, recall, and F1 under exact match and partial match NER evaluation. — Comparison of exact match and partial match NER evaluation metrics on the same predictions. Partial match awards credit for overlapping spans with correct entity types, while exact match requires perfect boundary alignment. The gap between the two schemes reflects boundary errors in the system's predictions, which inflate the apparent performance gap in real systems where boundary ambiguity is common.

The partial match scores are higher because they award credit for the boundary-error MONEY span that exact match rejects. In practice, this gap can be substantial (5-15 points of F1) on domains where entity boundaries are ambiguous.