Entity Linking and Knowledge Graph Construction › Evaluating Entity Linking

Entity-linking scores are comparable only under the stated evaluation protocol. A change in target spans, KB inventory, or matching rule can change what an F1 score measures. Check these choices before comparing systems:

Mention-level versus document-level. Mention-level micro metrics pool the counted decisions across documents, so documents with more gold or predicted mentions can contribute more to the aggregate. A document-macro metric instead averages a stated per-document measure. An all-mentions-correct document score is another, stricter criterion; it is not the definition of macro averaging. Report the aggregation rule and the handling of documents with no target mentions.

Strong versus weak matching. In GERBIL's annotation-matching terminology, strong matching requires the same entity and exact mention boundaries. Weak matching permits overlapping spans while still requiring the same entity (Röder et al., 2018). This is separate from resolving equivalent IDs or redirects to one referent. A subclass or topically related entity is not the same entity merely because it is nearby in a taxonomy.

NIL conventions. State whether NIL mentions are evaluated, whether distinct NIL mentions must also be clustered by referent, and how false accepted links and abstentions enter each denominator. An in-KB linking score alone does not measure NIL detection. Include both kinds of gold mentions when evaluating an open-KB system, and report the accepted-link precision and coverage when an application allows abstention.

Mention boundary errors. If the gold span is Apple Inc. and the detector returns only Inc., strict annotation matching rejects the prediction even if the ID is correct. Weak matching may accept the overlap. Record the span rule, tokenization, and offset convention so that a boundary disagreement is not mistaken for a referent disagreement.

Benchmark examples. AIDA-CoNLL adds entity disambiguation annotations to CoNLL newswire material (Hoffart et al., 2011). In the convention used by Kolitsas et al. (2018), AIDA-A is development data and AIDA-B is test data; do not tune on the latter. MedMentions provides a different setting, with manually annotated UMLS concept mentions in biomedical abstracts (Mohan and Li, 2019). Record the exact release, target ID space, mention inventory, and split files instead of assuming a benchmark name uniquely identifies the experiment.

Contamination and overlap. A target entity's presence in the inference KB is normally required for an in-KB linking task and is not, by itself, train-test leakage. Distinguish labeled training mentions of that entity, exposure to its description, reuse of test text, and development choices made after seeing test labels. These are different forms of overlap. An entity-disjoint evaluation such as ZESHEL addresses one transfer setting; a fixed-KB benchmark addresses another. State the intended setting before interpreting either score.

KB versioning also affects reproduction. Between KB snapshots, identifiers may be redirected or deprecated, records may be merged, and a conflated record may be split into separately identified records. Retain the mapping and snapshot used for evaluation rather than silently resolving historical annotations against a different live catalog. We discuss evaluation design further in Part LIII: Evaluation Fundamentals.

Decompose errors into mention detection failures, candidate generation failures (gold entity absent from the list), and ranking failures (gold entity retrieved but a different ID ranks first before acceptance). Evaluate NIL predictions against KB membership and assess the acceptance policy separately. An application can abstain even when the gold ID ranks first. Changing only a candidate-constrained reranker cannot recover an excluded gold entity. A low aggregate F1 alone does not identify which stage needs changing.