Entity Linking and Knowledge Graph Construction

A named entity recognizer might label Jordan as a person or a location. The tag alone does not identify the referent. To join the mention to a database record, you still need to decide whether it refers to Michael Jordan, the country, or another entity. Choosing the wrong record can attach otherwise correctly extracted facts to the wrong subject.

Recognition gives you a text span; linking assigns a knowledge-base identity. Strings can already be searched and joined, but equality of strings is an unreliable substitute for equality of referents. With correct links and distinct catalog IDs, different surface forms can contribute to the same entity record while identically spelled names for different entities remain separate.

Entity linking (EL) assigns text mentions to entries in a chosen knowledge base (KB). A linker can target Wikipedia articles, Wikidata items, biomedical concepts, or records in a private catalog. In this chapter, each catalog entry has an identifier and a description; aliases and graph relations provide additional evidence when available. These fields are useful design choices, rather than requirements that every knowledge base satisfies.

Knowledge graph construction (KGC) combines identity decisions with relation extraction and data management. Linking identifies the arguments. Relation extraction also needs a predicate, such as orbits in (mercury_planet, orbits, sun). A model can predict the arguments and predicate jointly or in separate stages. A construction workflow can validate candidates and resolve duplicate records under application rules while retaining supporting evidence. Provenance makes an assertion traceable; it does not establish that the source or the extractor was correct.

We study linking and construction together because their errors can propagate. A wrong linked argument can produce a wrong graph edge. In the other direction, graph compatibility can help disambiguate a mention, and an application constraint can flag an incompatible proposed edge. Such checks depend on the quality of the graph and the chosen validation rules.

Named entity recognition and BIO tagging from Named Entity Recognition and BIO Tagging provide a starting point for finding mention spans. Relation and event extraction from the preceding chapter provides candidate predicates and arguments. We reuse TF-IDF to explain context matching, then add entity identity and graph compatibility. The worked example compares a popularity baseline, local scoring, and joint scoring before building an inspectable assertion catalog.

Formally, let a document $d$ contain annotated entity mentions $M = \{m_1, \dots, m_n\}$. A mention has a half-open character span $[start_i,end_i)$ and a surface form $surf_i$. Given a fixed KB entity inventory $\mathcal{E}$, the target assignment is a total function $\Phi: M \to \mathcal{E} \cup \{\text{NIL}\}$. Each mention receives one entity identifier or the label NIL.

NIL means that the referent has no matching entry in the specified KB snapshot. It is distinct from uncertainty: a mention may have a KB referent even when the model cannot choose among candidates. An application may abstain and send such a case for review without claiming that the referent is absent. A nonentity span is a mention-detection error and should be rejected under the annotation policy, rather than automatically labeled NIL.

Three things make this hard.

Name variation. One entity can have several names or abbreviations. A catalog might record Apple Inc., Apple, and a translated name as aliases of the same entry. The exact accepted aliases depend on the catalog and the task. Storing only a canonical title misses any alternative spelling that exact lookup cannot normalize to that title.

Name collision. A string such as Mercury can name different entities. Normalization that helps retrieve spelling variants can also collapse distinctions between names. A string match can retrieve candidates. When the string is ambiguous, use context or other evidence before accepting a link, or abstain.

Unlinkable mentions. A document may refer to a newly created organization or another entity outside the KB. The proportion of these mentions depends on the corpus and KB snapshot. Evaluation must include them when the intended deployment permits out-of-KB referents.

For this chapter, group the decisions into four stages so their errors can be evaluated separately:

1. Mention detection: find the spans that refer to entities.
2. Candidate generation: propose a small set of KB entities that the mention could refer to.
3. Disambiguation (ranking): score candidates and pick one.
4. NIL prediction and acceptance policy: distinguish an out-of-KB referent from a link that the application will not accept.

The stages have different error sets. Detection determines which spans reach the linker, retrieval determines which IDs a restricted reranker can choose, and ranking orders those IDs. NIL prediction concerns KB absence; an acceptance policy may also abstain on an uncertain in-KB case. Measure the stages separately to identify which decisions an improved component could repair.

For a reranker restricted to a retrieved candidate list, a missing gold entity cannot be selected. On a fixed set of gold in-KB mentions, linking accuracy is therefore no greater than candidate recall@k. Every correct prediction requires the gold entity to have been retrieved, so the inequality follows by counting those cases. This bound does not directly cover mention-detection F1, NIL classification, or systems that generate additional candidates after retrieval. Measure these stages separately.

An alias dictionary can reveal ambiguity before a model is trained: inspect how many entities each string retrieves and how its anchor counts are distributed. Frequency and ambiguity are different properties. A rare name may have few training examples and still collide with another entity; a frequent name may have one dominant referent. Analyze both the number of candidates and the amount of supervision for the target entity.

The most frequent sense (MFS) baseline chooses the entity with the largest training-corpus anchor count for an alias. It exposes how much accuracy a system obtains from the alias prior alone. Its strength depends on the distribution of senses in the test set. In the toy table below, the company receives more apple counts than the fruit; those numbers are invented, not a measurement of Wikipedia anchors.

MFS is wrong whenever the current referent is outside the set of highest-count candidates. Words such as metal can support the element reading of Mercury, while Chicago Bulls can support Michael Jordan. Domain labels alone do not determine either answer: a chemistry article can discuss a planet, and a sports article can discuss a country. This is the same tension between a corpus prior and local evidence discussed in Lexical Semantics and Word Senses.

The prior estimates alias usage in its reference corpus; context supplies evidence about this occurrence. Neither establishes truth by itself. Compare a prior-only baseline with contextual scoring on development data from the intended domain, and inspect errors on rare entities separately. Botha et al. (2020) report challenges for rare entities and low-resource languages in their multilingual setting.

An NER model from Named Entity Recognition can supply candidate spans, but its annotation inventory may differ from the linking task's inventory. A linker that must identify nominal concepts, for example, needs spans that a proper-name tagger may never propose. Check the target annotation policy before reusing the detector unchanged.

The desired mention inventory depends on the linking task. It may include:

• Nominal mentions: "the company", "the tech giant", "the city".
• Nested and overlapping spans: in Bank of America Tower, the building and the bank are different entities.
• Targets outside a particular detector's inventory, possibly including works of art, laws, proteins, chemical compounds, or events.

For example, resolving "the company" can require discourse information beyond a name dictionary. The string Bank of America Tower contains an organization name, but whether to annotate both spans is a dataset decision, not an automatic requirement. Biomedical concept linking can use a different inventory again; MedMentions annotates mentions of UMLS concepts in biomedical abstracts (Mohan and Li, 2019).

One design is to propose spans at high recall and use later scoring to reject unsupported proposals. Scoring more proposals adds work when cost per proposal is fixed and positive; high detection recall alone does not guarantee precision after filtering. Keep two rejection reasons separate: a span may not be an entity mention at all, or it may be a valid mention whose referent is outside the KB. Evaluate span rejection and NIL handling according to the task's annotation rules.

Joint training offers another design: use mention-entity compatibility to help decide which spans to retain. Kolitsas et al. (2018) learn scores useful to both detection and disambiguation. A separate detector can also use KB-aware features; the distinction is whether its decisions are trained jointly with the linker, not whether it can access such evidence at all.

In a dictionary-based linker, the table provides both candidates and a prior. Record the corpus and KB versions used to build it, resolve redirects consistently, and choose normalization rules that match the task. A dictionary is one candidate-generation design; description-based retrieval can operate without an anchor table, as in the zero-shot setting of Logeswaran et al. (2019).

Candidate generation is a retrieval problem. Given a mention and its context, return up to $k$ plausible KB entities. Large inventories make the retrieval design consequential: the BLINK study evaluates retrieval over 5.9 million candidate entities (Wu et al., 2020). A cheap retrieval stage can reduce how many mention-description pairs an expensive reranker must evaluate. Those paper-specific counts are not current catalog-size claims.

If the reranker scores each returned candidate once, each additional candidate adds one pair evaluation. With fixed positive cost per pair, it also adds that scoring cost. For a fixed ranked retrieval result, enlarging $k$ cannot remove a retrieved gold entity, so recall@k is nondecreasing. It need not reach one: some referents may remain unretrieved at the tested depths. Select $k$ by measuring retrieval recall, downstream accuracy, and latency rather than assuming a particular list size is sufficient.

The classic approach builds an alias table from Wikipedia anchor text. For each surface string $m$, you store the set of entities it has been used to link to, along with counts. At inference time, you look up the normalized surface form and get candidates plus the prior

where:

• $P(e \mid m)$: the fraction of recorded reference-corpus links with normalized alias $m$ whose target is $e$
• $\text{count}(m \to e)$: how many times the alias table records the surface string $m$ being linked to entity $e$
• $\mathcal{C}(m)$: the candidate set for $m$, all entities $m$ has been linked to
• $\sum_{e' \in \mathcal{C}(m)} \text{count}(m \to e')$: the total link count over all candidates, the normalizing constant

Here $m$ denotes a normalized alias string, rather than the entire occurrence with its context. The denominator must be positive. We retain only positive-count candidates in this example, so their priors sum to one and the logarithms used later are defined. An unseen alias needs another retrieval route or an explicit fallback; a zero denominator is not a probability distribution.

Exact dictionary lookup is limited to the stored keys and candidates. Potential gaps include:

• Unseen aliases: a nickname, abbreviation, or translation absent from the collected dictionary.
• Case and morphology: APPLE, apples, Apple's.
• New entities: anything added to the KB after the dictionary was built.

If a shorthand has no dictionary key and normalization does not map it to one, exact lookup returns no candidates. Case and possessive handling likewise depend on the lookup rules: matching Apple's to apple needs an appropriate normalization or matching policy. A fixed table cannot return an entity missing from that table, even if a newer KB snapshot contains it.

The normalization rules from Text Normalization can reduce some spelling variation. Character n-gram retrieval is another option: overlapping character sequences can give similar spellings a nonzero retrieval score. This differs from the learned merge rules in Byte Pair Encoding. Measure whether approximate string matching retrieves useful variants and how many unrelated names it adds; spelling similarity alone does not establish identity.

A learned retriever can replace or augment the dictionary by mapping mention contexts and entity descriptions into a shared vector space. Similarity search then permits retrieval without an exact name match. This representation can connect different wording, although whether it retrieves the intended entity is an empirical question. The same retrieval pattern appears later in Retrieval-Augmented Generation.

BLINK (Wu et al., 2020) uses a bi-encoder to represent mention context and entity descriptions separately. Entity vectors can be precomputed, and an inner-product search retrieves candidates for a newly encoded mention. A cross-encoder then jointly encodes the mention and each retrieved description for reranking. The paper evaluates the resulting accuracy-speed tradeoff; a cross-encoder's advantage on another corpus must still be measured.

Description-based linking can represent an entity that has no labeled mention examples in linker training. In this bi-encoder design, encode its description and add the vector to the entity index; that can happen before queries arrive. This permits index updates without learning a new independent classifier weight for every entity. It does not guarantee retrieval: descriptions, language coverage, and domain shift still affect the match.

GENRE (De Cao et al., 2021) performs autoregressive entity retrieval: it generates an indexed entity name token by token, conditioned on the context. A trie can constrain continuations to allowed entity names. With correct termination constraints and a complete name-to-ID mapping, a completed output belongs to that index. It can still identify the wrong entity. The trie can cover the full name index or a supplied candidate subset, as in the paper's disambiguation experiments. Searching the full index avoids a separate retrieved shortlist, but remains limited by index coverage, the learned scores, and approximate decoding search.

Autoregressive decoding makes successive token decisions. Bi-encoder retrieval instead encodes the mention and searches a precomputed entity index. Their total latency also depends on batching, hardware, index search, name length, and decoding width. A valid constrained name is a structural guarantee, not a guarantee of better accuracy or of lower latency than another architecture.

The local scorer below combines the alias prior with context-description similarity. The prior summarizes the reference corpus's linked uses of that alias. Context can distinguish candidates when their descriptions overlap differently with the surrounding words, but it may provide little useful evidence. Neither feature guarantees the correct referent. Their contributions depend on the scoring rule and its weights.

$P(e \mid m)$ estimates how the reference corpus uses an alias. It favors whichever candidates receive more counts in that corpus, which need not be the intended referent in a new domain. A company-name prior estimated from technology reporting, for example, might differ from one estimated from a supplier catalog. Neither source establishes which reading fits a particular sentence.

Keep the prior-only argmax as a baseline. Adding context is useful only if it improves the decisions the application needs, so compare the combined ranker against that baseline on representative labeled examples.

Context can supply disambiguating evidence. Mercury orbits the Sun every 88 days shares orbits, 88, and days with the planet description below, whereas those tokens are absent from the element description. This overlap can favor the planet without the sentence saying "planet" or "element." Tokenization can change the similarity. Feature weighting can also change it by shifting a vector's direction. The prior contributes to the local score, and other candidates can change the ranking.

The classical implementation is a bag-of-words comparison. Represent the mention context (a window of surrounding tokens, or the whole sentence, or the whole document) as a sparse vector, represent each candidate entity by its KB description or by the text of its Wikipedia article, and take cosine similarity:

where:

• $\mathbf{c}_m$: the context vector for mention $m$ (for example, term counts or TF-IDF weights over the surrounding window)
• $\mathbf{d}_e$: the description vector for candidate entity $e$, built the same way from its KB description or article text
• $\mathbf{c}_m \cdot \mathbf{d}_e$: the dot product, summing the weighted overlap of shared terms
• $\lVert \mathbf{c}_m \rVert$ and $\lVert \mathbf{d}_e \rVert$: the Euclidean norms of the two vectors, normalizing away length differences

The vectors can use raw counts or TF-IDF weights. For nonzero real vectors, cosine lies in $[-1,1]$; for the nonnegative count vectors here it lies in $[0,1]$. The displayed ratio is undefined for a zero vector, so the code adopts a similarity of zero when either set is empty. IDF gives greater weight to terms found in fewer documents, but a rare word is not automatically relevant evidence for this mention. A term occurring in every description can still differ in frequency or normalized weight; ubiquity alone does not establish that every scoring scheme ignores it.

Explicit Semantic Analysis (Gabrilovich and Markovitch, 2007) represents text using Wikipedia-derived concepts rather than only word dimensions. A shared concept dimension can connect texts without a shared surface word. Applying that representation to linking would still require constructing the concept associations and evaluating the resulting mention-entity score.

Yamada et al. (2016) jointly embed words and entities in one vector space. Description-based bi-encoders and cross-encoders provide other ways to score mention-entity compatibility. A cross-encoder processes each pair jointly, while a bi-encoder can reuse separately encoded entity vectors. Pair evaluations can be batched; the relevant cost is the amount of pair computation, not a claim that each pair must occupy its own unbatched model call.

We use the following illustrative local score:

where:

• $s(e, m)$: the local score of candidate entity $e$ for mention $m$
• $P(e \mid m)$: the empirical probability of target $e$ among recorded links with normalized alias $a(m)$ in the reference corpus. In this score, $P(e \mid m)$ abbreviates $P(e \mid a(m))$ for mention occurrence $m$
• $\mathbf{c}_m$: a vector representation of the mention context (surrounding window, sentence, or document)
• $\mathbf{d}_e$: a vector representation of the candidate entity's description or article text
• $\cos(\mathbf{c}_m, \mathbf{d}_e)$: the cosine similarity between context and entity description
• $\alpha$: weight on the log prior probability of the candidate entity given the alias in the reference corpus
• $\beta$: weight on the context-similarity (local evidence) term

We restrict $\alpha$ and $\beta$ to finite nonnegative values. For an evaluated system, select their values on development data; the toy uses hand-chosen weights to expose the mechanics. The weights depend on feature scale as well as usefulness. For positive priors, the difference between their contributions is $\alpha\log(P(e_1\mid m)/P(e_2\mid m))$. The logarithm $\log p$ tends to negative infinity as $p$ approaches zero from above. The weighted contribution $\alpha\log p$ has that limit when $\alpha>0$; with $\alpha=0$ it is zero for every positive $p$. Logarithms do not by themselves prevent popularity from dominating context. The toy candidates all have positive priors.

Other features can include type compatibility, name similarity, and graph proximity to already linked entities. If a mention is known to denote a person under the annotation policy, a country candidate is incompatible. A predicted NER type can itself be wrong, however, and metonymic uses depend on the annotation convention. Use type predictions as fallible evidence, or justify any hard constraint with the task's rules.

The weighted local score is a ranking quantity. Its value of 0.9 would not imply a 90 percent chance of correctness. Interpreting a score probabilistically requires a suitable calibration procedure and validation on the relevant distribution. Margins can support an acceptance policy, but need their own evaluation.

Independent local decisions can miss evidence available elsewhere in a document. Once Chicago Bulls is linked to the basketball team, its connection to Michael Jordan can help rank an ambiguous Jordan mention. This is a useful signal, not a universal topical constraint: documents can compare unrelated people, countries, products, or events.

Collective entity linking scores combinations of assignments rather than choosing every entity independently. Ratinov et al. (2011) study local and global disambiguation, and Hoffart et al. (2011) combine mention-entity compatibility with coherence among entities. We use a small pairwise objective to inspect that idea. Whether coherence improves accuracy depends on the data, the compatibility measure, and its weight.

Milne and Witten (2008) use Wikipedia link structure to compare entities. The same shared-inlink count can represent very different degrees of overlap for small and large inlink sets. The following normalized link-overlap score accounts for those sizes; we explicitly distinguish its distance form from the compatibility reward used in our maximization problem.

Let $W$ be the article universe, with more than one article, and let $A$ and $B$ be the sets of articles linking to entities $a$ and $b$. For nonempty overlap and $\min(|A|,|B|)<|W|$, first compute a normalized link distance and then turn it into a bounded compatibility reward:

where:

• $W$: the article universe used by the link graph
• $A$ and $B$: articles linking to $a$ and $b$, respectively
• $|A\cap B|$: their number of shared inlink articles
• $D(a,b)$: a distance, with smaller values indicating greater overlap at fixed marginal counts
• $r(a,b)$: the clipped compatibility reward used below

We use natural logarithms; any common valid logarithm base gives the same ratio. Identical nonempty proper subsets give $D=0$ and $r=1$. At fixed $|A|$, $|B|$, and $|W|$, more shared articles decrease the distance. The clipped reward cannot decrease, but can remain zero while $D\geq1$. This normalization is not a probability or a statistical test of independence.

We explicitly return $r=0$ when either inlink set is empty or the sets are disjoint. If both sets equal $W$, the ratio has a zero denominator; the toy implementation returns $r=1$ for that identical-set case. Clamping $1-D$ below at zero is an implementation convention because the unbounded distance can exceed one. These boundary policies make the score defined on every pair in our small graph.

Let $\mathcal{C}(m_i)$ be the finite, nonempty candidate set for each mention $m_i$ included in the joint assignment. Apply a retrieval fallback or abstain on empty candidate lists before forming the product. With finite scores, the linking objective is a global optimization over those mentions.

where:

• $\hat{\mathbf{e}}$: the joint assignment of one entity to each mention, i.e. a tuple $(e_1, \dots, e_n) \in \mathcal{C}(m_1) \times \cdots \times \mathcal{C}(m_n)$
• $\mathcal{C}(m_i)$: the candidate entity set for mention $m_i$
• $s(e_i, m_i)$: the local score of entity $e_i$ for mention $m_i$ (prior plus context similarity)
• $r(e_i, e_j)$: the pairwise relatedness between entities $e_i$ and $e_j$ (Milne-Witten, defined above)
• $\gamma\geq0$: a finite weight on the pairwise coherence term, trading off document coherence against local evidence

This is a structured prediction score with unary terms $s$ and pairwise terms $\gamma r$. When every candidate set is nonempty and all scores are finite, exponentiating the total score and normalizing over the finite Cartesian product defines a conditional distribution. Its maximum-probability assignments are exactly the highest-scoring assignments. In that interpretation the displayed terms are log-potentials. The factorization includes every mention pair, unlike the linear-chain model in Conditional Random Fields.

For arbitrary fully connected pairwise terms, the linear-chain decomposition used by the Viterbi Algorithm does not directly solve this objective. Exhaustive search considers the full Cartesian product of candidate sets. Special parameter choices can be simpler, including $\gamma=0$ below. Options for the general case include:

• Exhaustive search when the candidate-set product is small enough that you can score every assignment.
• Integer linear programming with one binary choice variable per (mention, candidate), one-of-$k_i$ constraints for each mention $m_i$ with $k_i = |\mathcal{C}(m_i)|$, and auxiliary variables or equivalent constraints for pairwise products. These are assignment constraints, not transitivity constraints between mentions. AIDA's graph algorithm instead searches for a dense subgraph (Hoffart et al., 2011).
• Approximate search with loopy max-product belief propagation, or with Gibbs sampling from the score-induced distribution and selection of the highest-scoring complete assignment visited.
• Greedy sequential refinement: visit each mention, test its candidates while holding the other assignments fixed, and accept the best change only if it strictly improves the score. Repeat until a complete pass makes no change. On a finite candidate space, strict improvements cannot revisit an assignment. The resulting assignment has no improving single-mention move, but need not be the global optimum.

At $\gamma=0$, the objective decomposes into independent local choices, provided each candidate set is nonempty. Increasing $\gamma$ gives compatibility more influence and can overturn local evidence, including a correct local decision. There is no guarantee that a news article benefits from a larger value than another document. Select the weight on held-out development data and inspect mixed-topic documents for erroneous pressure toward one topic.

Our objective permits coherence to override local evidence; it does not enforce a tie-breaker-only policy. A different design could gate that contribution by a confidence criterion, but such a gate would need its own definition and validation. Keep this distinction visible when interpreting a successful toy example: a useful compatibility bonus can also propagate a wrong neighboring assignment.

The stage decomposition helps diagnose errors, but it does not require separate learned representations. Components may share an encoder or be trained jointly. Joint systems can use linking evidence when deciding whether a span is a mention, while a modular design permits separate replacement or evaluation of its components.

Kolitsas et al. (2018) jointly learn mention detection and disambiguation scores for candidate span-entity pairs. ELQ (Li et al., 2020) uses a bi-encoder and predicts mention starts, ends, and token membership from question representations. Its mention representations are compared with entity vectors. The paper's one-pass description concerns the input question's BERT encoding; entity encoding and index search are separate parts of the computation. Sharing that question representation permits detection and linking to use the same contextual evidence. Evaluate total latency rather than inferring it from the number of question-encoder passes alone.

Joint mention detection does not by itself require a fixed entity inventory. The update procedure depends on the entity representation. A model with entity-specific learned parameters may need training changes, whereas a description-based model may encode new descriptions and update its index. Distinguish model retraining from index rebuilding when comparing maintenance costs.

Zero-shot entity linking evaluates entities without labeled training mentions in the specified split. Description-based matching gives the model information about a new entity without requiring a learned class weight for its ID. State what was held out: entities, domains, languages, and mention strings are different generalization tests. This use of "zero-shot" also differs from prompting an LLM without task demonstrations.

ReFinED (Ayoola et al., 2022) combines mention detection, fine-grained entity typing, and disambiguation in one document forward pass, using entity types and descriptions. Its reported efficiency and accuracy apply to the paper's benchmarks and configurations. This is an example of sharing computation across predicted mentions, not a guarantee that every target mention in a new document will be detected correctly.

ZESHEL (Logeswaran et al., 2019) evaluates linking to unseen entities in held-out domains using entity descriptions, without in-domain labeled training data or alias metadata in its task formulation. It tests a relevant kind of transfer, but does not establish performance on an arbitrary private or continually changing KB. A deployment evaluation should reproduce the intended languages, entity descriptions, mention inventory, and unknown-entity rate.

Botha et al. (2020) use a dual encoder for mentions in more than 100 languages, linking to a language-agnostic KB. This is distinct from mGENRE, the autoregressive multilingual system of De Cao et al. (2022). mGENRE predicts entity names and considers names in multiple languages, rather than fixing every target name to one canonical language. Both approaches separate the entity's identity from the language of a particular mention or name. The outline for Multilingual Language Models and Cross-Lingual Transfer lists shared representations under Cross-Lingual Transfer and Alignment.

Multilingual linking adds possible gaps in aliases, descriptions, and training supervision across languages. A cross-lingual representation may match a mention to a description in another language, but that ability must be evaluated. A missing description alone does not establish that the entity is absent from the KB: another retrieval route may find it, or the system may abstain.

An LLM can also be placed after retrieval: supply the mention, its context, and candidate IDs with descriptions, then request one ID or an abstention action. This is a system design option, not an accuracy claim. Evaluate it against a smaller reranker and the prior-only baseline on the same candidate lists, with the same mention spans and KB snapshot.

Evaluate the following potential failure modes:

• Identifier hallucination. An unconstrained output can contain an invented QID or title. Restrict the response to an allowed candidate ID or the designated abstention action. Check any returned ID against the chosen KB snapshot.
• Popularity effects. Test whether the reranker improves rare-entity cases rather than relying on the familiar candidate name.
• Prompt sensitivity. Evaluate several fixed candidate permutations and record whether the decisions change. Testing one fixed order does not test invariance to other orders.
• Cost and latency. Measure total cost at the intended mention volume, including retrieval, batching, output validation, and any fallback calls.

Validate every returned ID against the allowed list and the chosen KB snapshot. This can reject an invented identifier; it cannot detect a semantically wrong choice that is nevertheless a valid candidate. Likewise, a fixed candidate order does not establish the absence of popularity or order bias. Keep identifier validity, referential correctness, and run-to-run variability as separate checks.

A closed-KB disambiguation benchmark may assume every gold mention has a KB referent. An open-KB task relaxes that assumption and needs NIL evaluation. Independently of the gold label, an application may abstain when evidence is insufficient. Report which setting the system and its metrics implement.

Possible abstention features include:

• The top-ranked candidate’s score $s(e^*, m)$.
• The margin $s(e^*, m) - s(e^{(2)}, m)$ between the top two candidates.
• The similarity assigned to the top-ranked candidate by the retriever, when the retriever supplies one.

These features indicate uncertainty under the scorer, not proof of KB absence. A low margin may mean two valid KB candidates are difficult to distinguish. An empty candidate list may mean an alias lookup failed for a known entity. To predict NIL itself, validate the decision rule against both in-KB and out-of-KB gold mentions. A threshold can shift when the corpus, candidate generator, or KB changes.

An acceptance policy should reflect error costs. If false accepted links are more costly than delayed assertions, abstention may be preferable to forcing a candidate. Rejecting a link that would otherwise be accepted lowers accepted coverage for a fixed mention set; it does not guarantee a correct remaining graph. Mislabeling an in-KB referent as NIL can also create duplicate local nodes, so record abstention separately when the evidence establishes uncertainty rather than absence.

Fine-grained entity typing predicts categories such as person/artist/musician, with the hierarchy determined by the chosen type inventory. ReFinED uses fine-grained typing jointly with linking. Such predictions can contribute compatibility evidence, but a predicted type should not be silently treated as a verified ontology constraint. Application validation rules must also specify how they handle missing or uncertain types.

Entity-linking scores are comparable only under the stated evaluation protocol. A change in target spans, KB inventory, or matching rule can change what an F1 score measures. Check these choices before comparing systems:

Mention-level versus document-level. Mention-level micro metrics pool the counted decisions across documents, so documents with more gold or predicted mentions can contribute more to the aggregate. A document-macro metric instead averages a stated per-document measure. An all-mentions-correct document score is another, stricter criterion; it is not the definition of macro averaging. Report the aggregation rule and the handling of documents with no target mentions.

Strong versus weak matching. In GERBIL's annotation-matching terminology, strong matching requires the same entity and exact mention boundaries. Weak matching permits overlapping spans while still requiring the same entity (Röder et al., 2018). This is separate from resolving equivalent IDs or redirects to one referent. A subclass or topically related entity is not the same entity merely because it is nearby in a taxonomy.

NIL conventions. State whether NIL mentions are evaluated, whether distinct NIL mentions must also be clustered by referent, and how false accepted links and abstentions enter each denominator. An in-KB linking score alone does not measure NIL detection. Include both kinds of gold mentions when evaluating an open-KB system, and report the accepted-link precision and coverage when an application allows abstention.

Mention boundary errors. If the gold span is Apple Inc. and the detector returns only Inc., strict annotation matching rejects the prediction even if the ID is correct. Weak matching may accept the overlap. Record the span rule, tokenization, and offset convention so that a boundary disagreement is not mistaken for a referent disagreement.

Benchmark examples. AIDA-CoNLL adds entity disambiguation annotations to CoNLL newswire material (Hoffart et al., 2011). In the convention used by Kolitsas et al. (2018), AIDA-A is development data and AIDA-B is test data; do not tune on the latter. MedMentions provides a different setting, with manually annotated UMLS concept mentions in biomedical abstracts (Mohan and Li, 2019). Record the exact release, target ID space, mention inventory, and split files instead of assuming a benchmark name uniquely identifies the experiment.

Contamination and overlap. A target entity's presence in the inference KB is normally required for an in-KB linking task and is not, by itself, train-test leakage. Distinguish labeled training mentions of that entity, exposure to its description, reuse of test text, and development choices made after seeing test labels. These are different forms of overlap. An entity-disjoint evaluation such as ZESHEL addresses one transfer setting; a fixed-KB benchmark addresses another. State the intended setting before interpreting either score.

KB versioning also affects reproduction. Between KB snapshots, identifiers may be redirected or deprecated, records may be merged, and a conflated record may be split into separately identified records. Retain the mapping and snapshot used for evaluation rather than silently resolving historical annotations against a different live catalog. We discuss evaluation design further in Part LIII: Evaluation Fundamentals.

Decompose errors into mention detection failures, candidate generation failures (gold entity absent from the list), and ranking failures (gold entity retrieved but a different ID ranks first before acceptance). Evaluate NIL predictions against KB membership and assess the acceptance policy separately. An application can abstain even when the gold ID ranks first. Changing only a candidate-constrained reranker cannot recover an excluded gold entity. A low aggregate F1 alone does not identify which stage needs changing.

Linked arguments give a relation extractor candidate endpoints; they do not turn a sentence into a verified fact. Graph construction adds choices about identity, relations, validation, and storage. Consistent IDs let assertions from different documents join, while source records let a consumer inspect how each assertion arose.

A graph construction project should decide which of the following it needs:

• Nodes with stable identity. The entity mercury_planet must be one node, referenced consistently across documents, not a new node per mention.
• Edges with a schema. This project can use a controlled relation vocabulary: orbits, is_capital_of, member_of. Free-text relation labels alone do not specify validation constraints or equivalence rules.
• Types or a class hierarchy when the application needs type-aware retrieval or inference. A hierarchy alone does not infer geographic containment without the relevant relations and rules.
• Provenance fields chosen for the application, such as a source document, supporting span, extractor version, and extraction time.
• Temporal scope for relations that change, such as (X, ceo_of, Y), including a representation for unknown or open-ended bounds.
• Confidence or evidence counts when they support review or fusion. Repeated copies of one source are not independent corroboration. A count alone does not establish which conflicting assertion is correct.

These choices support different operations. Stable identifiers support joins, but do not automatically prevent duplicates. A controlled relation vocabulary reduces predicate variation; application validation rules can detect incompatible records. Provenance supports tracing assertions, temporal scope supports date-aware queries, and scores can prioritize review. None of these is a guarantee of correctness, and a class hierarchy or a confidence field is not a defining requirement for every knowledge graph.

Two data models worth distinguishing are RDF and property graphs.

An RDF 1.1 graph is a set of subject-predicate-object triples. Subjects can be IRIs or blank nodes; predicates are IRIs; objects can be IRIs, blank nodes, or literals. In the following illustrative triples all three positions use prefixed IRIs. Here ex: abbreviates the example namespace https://example.org/.

@prefix ex: <https://example.org/> .

ex:mercury_planet ex:orbits ex:sun .
ex:amman ex:is_capital_of ex:jordan_country .

An RDF dataset contains a default graph and zero or more named graphs (W3C RDF 1.1 Concepts, 2014). An application can group assertions by source or context using graph names, but must define what that grouping means. Statement-level metadata can instead use a statement resource, including RDF 1.1 reification. Reifying a triple describes it; it does not by itself assert the described triple. SPARQL provides standardized graph queries, including a federation extension, but we do not implement that query language in the toy store below.

Property graphs attach properties to vertices and edges, so an application can store source and score fields directly on an edge. They are not an area without standards: ISO/IEC 39075:2024 defines GQL data structures and operations for property graphs. An application still needs to choose its entity identifiers, source policy, and domain ontology. Interoperability in either model still requires agreement about those meanings.

Choose a representation according to the required interchange formats, query behavior, inference rules, and update workflow. RDF and a property graph can both carry provenance when it is modeled explicitly. A data-model label alone does not establish faster queries, better identity resolution, or more trustworthy assertions.

A realistic construction pipeline looks like this:

1. Ingest documents and record source metadata (URL, retrieval time, license).
2. Segment and parse into sentences and structured blocks.
3. Detect mentions, retrieve and rank KB candidates, and record separate NIL predictions or abstentions.
4. Extract relations with patterns, classifiers, or an LLM, conditioned on the linked entity types.
5. Canonicalize arguments to nodes under an explicit merge policy, including local nodes for justified out-of-KB referents.
6. Validate against schema constraints: type checking, cardinality, domain/range restrictions.
7. Compare candidates with existing content, retaining conflicts that the fusion policy cannot justify resolving.
8. Store surviving assertions with the provenance captured during ingestion and extraction.

Capture provenance when candidates are created and preserve it through validation and fusion; do not wait until the last step to reconstruct missing evidence. Linking, canonicalization, and fusion errors require different repairs. Evaluate each with appropriate labeled cases as well as testing the complete pipeline.

Entity linking maps text mentions to KB entries. Entity resolution identifies records or nodes that refer to the same entity; a later merge policy decides how to combine their fields and identifiers. Resolution can use names, attributes, graph neighborhoods, and source documents when those are retained. Its available context depends on the record schema and retained evidence; record matching need not discard the source text.

The mechanics are similar to linking but operate on records rather than mentions:

• Blocking. All-pairs comparison of $n$ records requires $n(n-1)/2$ unordered pairs. At one million records, that is 499,999,500,000 pairs. When that cost exceeds the available budget, blocking can retrieve a smaller candidate set using keys such as normalized names, identifiers, or neighbors.
• Pairwise matching. Score each candidate pair with string similarity (Jaro-Winkler, token-set ratio), attribute agreement, and neighborhood overlap. Then classify match/non-match.
• Clustering. Decide which candidate records belong to one entity. Connected components enforce a transitive grouping; a clustering objective can instead penalize inconsistent pair decisions. Choose a representative only after deciding membership.

Blocking trades candidate coverage against comparison cost. A pair excluded by every retrieval or blocking route will not reach the pairwise matcher. Candidate keys can use name tokens, identifiers, or neighbors, but their usefulness depends on missingness, name collisions, and graph errors. Measure candidate-pair recall on labeled duplicates rather than assuming one kind of key is best.

A pairwise classifier can say that A matches B and B matches C while rejecting A-C. Connected components merge all three because a path connects them. One alternative is a clustering objective that penalizes both joining dissimilar records and separating similar ones; this still depends on the quality of the pairwise evidence and its optimization. Choosing a canonical representative happens after deciding cluster membership. It does not repair a mistaken cluster.

owl:sameAs is appropriate when identity is the intended, justified claim. A custom property does not become weaker merely because it has another name: specify its semantics and the reasoning rules that use it. Likewise, recording a canonical ID can cause an application to merge records, so that decision also needs evidence and a reversible history.

A consumer may need to ask: where did this assertion come from, and when does it apply? Source provenance and temporal scope support different answers. Their required detail depends on the application; not every assertion needs a finite validity interval.

An old but accurately extracted source can describe a relation that no longer holds. Conversely, a current assertion can have weak source support. One timestamp cannot in general represent both when the relation applied and when the system recorded it. Keep those roles explicit when the query requirements distinguish them.

PROV-O provides classes and properties for representing entities, activities, agents, and their provenance relationships (W3C, 2013). For the extraction workflow here, useful record fields include:

• The source document identifier and, ideally, a content hash.
• The character span in the source that supports the assertion.
• The extractor name and version.
• The timestamp of extraction.
• A confidence score.

Retain the source version together with the span. Offsets into a mutable page can point to different text after an edit. A content hash identifies a version but does not recover its bytes, so keep a snapshot or another reliable retrieval route when reproduction matters. Record the offset convention, such as half-open Unicode character positions. The source text and locator let a reviewer inspect the asserted evidence; they do not prove that the assertion follows from it.

Some relations change over time, including employment and office holding. Wikidata statements can use qualifiers such as start and end time to record context. Its preferred, normal, and deprecated ranks are a separate mechanism. A normal rank is neutral about accuracy and currency, and several statements can have preferred rank. Rank alone does not determine whether a statement is valid on a requested date.

Wikidata's model is a good template even if you do not use RDF:

• Main claim: the item, property, and value; the statement record can also carry qualifiers, references, and rank.
• Qualifiers: additional context, such as time bounds, scope, or role.
• References point to sources that support a statement or document where its value came from.
• Rank: a preference or deprecation annotation, which can apply to several values.

Separating values, qualifiers, references, and ranks permits different assertions to be represented without collapsing their contexts. A consumer must still define how to handle disagreement, missing references, and temporal qualifiers. Neither references nor ranks are calibrated confidence probabilities.

Bitemporal modeling separates valid time, when an assertion applies in the modeled world, from transaction time, when it is recorded in the database (Jensen and Snodgrass, 2018). Suppose a fictional contract takes effect in 2020 but the database first records it in 2023. Its valid-time start is 2020 and its transaction-time start is 2023. The effective date must come from the evidence; an announcement date is not automatically an effective date. With the relevant retained history, the two axes distinguish a query about 2021 from a query about what was recorded as of 2022.

Before composing assertions, check that the relation-composition rule is sound for their meanings and that their temporal and other scopes align. Calibrated marginal probabilities alone do not determine a conjunction probability. Independence is sufficient for multiplying the marginals under the relevant probability model. With premise probabilities 0.9 and 0.8 but no dependence information, their conjunction can range from 0.7 to 0.8. A product score of 0.72 or a minimum score of 0.8 may instead be chosen as a heuristic, but must be labeled as such. A valid conjunction also does not establish an unsupported inference rule.

We will compare three linking strategies using a small KB, an alias table, eight documents, and a coherence graph. The data and weights are chosen to expose the scoring mechanics. These results are a worked calculation, not a benchmark or an estimate of deployment accuracy.

The toy KB has eleven entities. Each has a label and a short description, which is what a description-based linker would index:

The policy accepts the planet for d1 and abstains on d8 and the empty table. Mercury's authored gold label in d8 is the planet, which is in the toy KB. The margin policy abstains on this in-KB gold target. An empty lookup also cannot establish KB absence. We have not implemented an open-world NIL detector or changed the three-strategy comparison above.

We now start from hand-written relation-extraction candidates with linked arguments. The toy KB omits two objects, so we explicitly label those arguments NIL and assign document-scoped local nodes. This is a construction policy for this fixture; a failed link in general would first require separating uncertainty from confirmed KB absence.