BIO Tagging › BIO Evaluation and Comparison to Gold Standards
Language AI HandbookAll chapters BIO Tagging › BIO Evaluation and Comparison to Gold StandardsOne section · Sections in this chapter8 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Measuring NER system accuracy requires comparing predicted BIO sequences against gold-standard annotations. The standard evaluation protocol counts entities at the span level, not the token level. A prediction is correct only if both the entity boundaries and the entity type match the gold standard exactly.
This strict matching criterion has practical effects. Consider a gold annotation of "New York City" as LOC. A model that predicts B-LOC I-LOC for "New York" (missing "City") scores zero on this entity. Neither a partial credit nor a type-only credit is awarded. The full-span requirement means that boundary errors are penalized just as heavily as type errors. This motivates the attention to boundary precision we saw in the BIOES scheme discussion.
def evaluate_ner(gold_tokens, gold_tags, pred_tags):
"""
Evaluate NER predictions against gold standard.
Uses exact span matching: both boundaries and type must match.
Returns precision, recall, and F1 score.
"""
gold_spans = set(
(s, e, t) for s, e, t, _ in bio_to_spans(gold_tokens, gold_tags)
)
pred_spans = set(
(s, e, t) for s, e, t, _ in bio_to_spans(gold_tokens, pred_tags)
)
true_positives = gold_spans & pred_spans
false_positives = pred_spans - gold_spans
false_negatives = gold_spans - pred_spans
precision = len(true_positives) / len(pred_spans) if pred_spans else 0.0
recall = len(true_positives) / len(gold_spans) if gold_spans else 0.0
f1 = (
2 * precision * recall / (precision + recall)
if precision + recall > 0
else 0.0
)
return {
"precision": precision,
"recall": recall,
"f1": f1,
"tp": len(true_positives),
"fp": len(false_positives),
"fn": len(false_negatives),
"true_positives": true_positives,
"false_positives": false_positives,
"false_negatives": false_negatives,
}
# Gold and predicted sequences for evaluation example
eval_tokens = ["Barack", "Obama", "visited", "New", "York", "City", "yesterday"]
eval_gold = ["B-PER", "I-PER", "O", "B-LOC", "I-LOC", "I-LOC", "O"]
# Prediction 1: Perfect prediction
pred_perfect = ["B-PER", "I-PER", "O", "B-LOC", "I-LOC", "I-LOC", "O"]
# Prediction 2: Boundary error (misses last LOC token)
pred_boundary = ["B-PER", "I-PER", "O", "B-LOC", "I-LOC", "O", "O"]
# Prediction 3: Type error (predicts ORG instead of LOC)
pred_type_err = ["B-PER", "I-PER", "O", "B-ORG", "I-ORG", "I-ORG", "O"]NER Evaluation: Three Prediction Scenarios
============================================================
Perfect prediction:
Predicted: B-PER I-PER O B-LOC I-LOC I-LOC O
TP=2, FP=0, FN=0
Precision=1.00, Recall=1.00, F1=1.00
Boundary error (New York only):
Predicted: B-PER I-PER O B-LOC I-LOC O O
TP=1, FP=1, FN=1
Precision=0.50, Recall=0.50, F1=0.50
Missed spans: {(3, 6, 'LOC')}
Wrong spans: {(3, 5, 'LOC')}
Type error (LOC predicted as ORG):
Predicted: B-PER I-PER O B-ORG I-ORG I-ORG O
TP=1, FP=1, FN=1
Precision=0.50, Recall=0.50, F1=0.50
Missed spans: {(3, 6, 'LOC')}
Wrong spans: {(3, 6, 'ORG')}The evaluation results illustrate the strictness of span-level matching. A boundary error that misses just one token of a three-token entity scores F1=0.5, the same as a type error where the entire entity is correctly bounded but labeled as the wrong type. Both errors are completely wrong from the perspective of exact span matching. A system designer choosing between BIOES and BIO might find this motivating: if boundary errors are equally costly as type errors, the explicit end markers of BIOES, which help the model learn exact boundaries, can directly improve F1.