Information Extraction › Evaluation

Evaluating IE systems requires task-specific metrics. The evaluation philosophy is the same across all IE tasks: partial credit is not given. Either a prediction matches the gold annotation exactly, or it counts as an error.

For NER, a prediction is correct only if both the span boundaries and the entity type match exactly. The standard metrics are entity-level precision, recall, and F1. To understand why these are defined at the entity level rather than the token level, consider a system that correctly identifies the tokens "New" and "York" but labels them as two separate entities instead of one "New York" span: token-level accuracy would be high, but entity-level evaluation would mark this as an error because the correct span boundary is not reproduced.

Precision measures how many predicted entities are correct:

$$\text{Precision} = \frac{|\text{TP}|}{|\text{TP}| + |\text{FP}|}$$

Recall measures how many gold entities the model successfully predicted:

$$\text{Recall} = \frac{|\text{TP}|}{|\text{TP}| + |\text{FN}|}$$

The harmonic mean of precision and recall, F1, provides a single balanced score:

$$\text{F1} = \frac{2 \cdot \text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}$$

where:

• $|\text{TP}|$ is the count of True Positives: predicted spans that match a gold span in both token boundaries and entity type
• $|\text{FP}|$ is the count of False Positives: predicted spans with no matching gold span (incorrect prediction or wrong type)
• $|\text{FN}|$ is the count of False Negatives: gold spans that the model failed to predict entirely

The harmonic mean is used rather than the arithmetic mean because it penalizes imbalance between precision and recall more heavily. A system with precision 1.0 and recall 0.0 would have an arithmetic mean of 0.5 but an F1 of 0.0, correctly reflecting that the system is useless because it extracts nothing.

For relation extraction, evaluation is similarly span-based: a predicted triple (subject, predicate, object) is correct only when all three components match the gold annotation. This means RE F1 scores are always lower than NER F1 scores on the same document: a correct relation requires a correct subject entity AND a correct object entity AND a correct relation type. If the NER system has F1 of 0.90, the upper bound on RE F1 is approximately $0.90^2 \approx 0.81$, even with a perfect relation classifier. This compounding effect is why end-to-end IE pipelines are evaluated separately from their individual components, and why improvements in NER propagate through to RE.

For structured output, schema validation is a minimum bar (did the model produce valid JSON with all required fields?). Beyond that, field-level accuracy measures what fraction of fields are correctly populated, and document-level accuracy measures what fraction of complete records are perfect. Document-level accuracy is a demanding metric: a record with nine correct fields and one wrong field scores zero. For applications where a partially correct record is still useful (e.g., database entries that can be manually corrected), field-level accuracy is more informative.

The precision of 1.0 reflects that every predicted entity is correct on its own terms, but the recall and F1 drop because of the missed entity (Cupertino) and the misclassified type (California labeled as LOC instead of GPE). In practice, NER errors cluster around rare entity types, boundary cases (multi-word entities), and ambiguous surface forms. The confusion between LOC and GPE is characteristic: geographic locations that are also political entities are systematically difficult, and different annotation guidelines disagree on the correct label.

The following chart shows how NER errors are distributed across error types (false positives, false negatives, and type errors) at different system confidence thresholds, illustrating the precision-recall tradeoff in practice.