Entity Linking and Knowledge Graph Construction › Worked Example
Language AI HandbookAll chapters Entity Linking and Knowledge Graph Construction › Worked ExampleSection view · Sections in this chapter13 / 17 ↓
Reading tools Definitions, links & notes
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
We will compare three linking strategies using a small KB, an alias table, eight documents, and a coherence graph. The data and weights are chosen to expose the scoring mechanics. These results are a worked calculation, not a benchmark or an estimate of deployment accuracy.
The toy KB has eleven entities. Each has a label and a short description, which is what a description-based linker would index:
PythonIn [3]Expand
import math
import re
from collections import defaultdict
from itertools import product
import numpy as np
import pandas as pdPythonIn [4]Expand
ENTITIES = {
"mercury_planet": {
"label": "Mercury (planet)",
"desc": (
"Mercury is the smallest planet in the Solar System and closest to the Sun. "
"Mercury orbits the Sun every 88 days and has a heavily cratered surface."
),
},
"mercury_element": {
"label": "Mercury (element)",
"desc": (
"Mercury is a chemical element with the symbol Hg and atomic number 80. "
"It is a heavy silvery metal that is liquid at room temperature and was "
"once common in thermometers."
),
},
"mercury_god": {
"label": "Mercury (mythology)",
"desc": (
"Mercury is a god in Roman religion and mythology, the patron god of "
"financial gain, commerce, and eloquence. He is a messenger of the gods "
"and the Roman counterpart of the Greek god Hermes."
),
},
"apple_inc": {
"label": "Apple Inc.",
"desc": (
"Apple Inc. is an American multinational technology company headquartered in "
"Cupertino, California. Apple designs the iPhone, the Mac, and the iPad, and "
"runs the App Store and iCloud."
),
},
"apple_fruit": {
"label": "Apple (fruit)",
"desc": (
"An apple is an edible fruit produced by an apple tree and grown in orchards. "
"Ripe apples contain fiber and vitamin C. Harvest timing depends on cultivar and climate."
),
},
"jordan_country": {
"label": "Jordan",
"desc": (
"Jordan is a country in Western Asia, bordered by Saudi Arabia, Iraq, Syria, "
"and Israel. The capital city is Amman."
),
},
"jordan_person": {
"label": "Michael Jordan",
"desc": (
"Michael Jordan is an American former professional basketball player and "
"businessman. He played fifteen seasons in the National Basketball Association "
"and won six championships with the Chicago Bulls."
),
},
"chicago_bulls": {
"label": "Chicago Bulls",
"desc": (
"The Chicago Bulls are an American professional basketball team based in "
"Chicago. The Bulls compete in the National Basketball Association."
),
},
"amman": {
"label": "Amman",
"desc": (
"Amman is the capital and largest city of Jordan. It is a commercial "
"and cultural center of Jordan."
),
},
"iphone": {
"label": "iPhone",
"desc": (
"The iPhone is a line of smartphones designed and marketed by Apple. It runs "
"the iOS operating system."
),
},
"hubble": {
"label": "Hubble Space Telescope",
"desc": (
"The Hubble Space Telescope is a space telescope launched into low Earth "
"orbit in 1990. It observes planets, stars, and distant galaxies."
),
},
}Notice that the descriptions are written in natural language and contain the kind of distinguishing terms a context-similarity scorer would look for. The planet description mentions planet, Sun, 88 days, and cratered. The element description mentions chemical element, Hg, metal, liquid, and thermometers. The god description mentions Roman, mythology, commerce, and Hermes. After tokenization and masking, shared terms contribute to the overlap. All retained description terms contribute to the norm.
Next, the alias table. These invented counts stand in for anchor statistics. Their proportions let us inspect cases where context or coherence changes the prior's preferred candidate; they do not estimate Wikipedia usage.
PythonIn [5]Expand
ALIAS_COUNTS = {
"mercury": {
"mercury_element": 450,
"mercury_planet": 400,
"mercury_god": 150,
},
"apple": {"apple_inc": 650, "apple_fruit": 350},
"jordan": {"jordan_country": 700, "jordan_person": 300},
"chicago bulls": {"chicago_bulls": 1000},
"amman": {"amman": 1000},
"iphone": {"iphone": 1000},
"hubble": {"hubble": 1000},
}
ALIAS_TO_CANDIDATES = {
alias: list(counts) for alias, counts in ALIAS_COUNTS.items()
}The alias table supplies both the priors and the candidate sets. mercury has three candidates; apple and jordan have two each; the other listed aliases have one each. A production dictionary also needs a policy for normalization, unseen aliases, redirects, and updates.
The skew in the alias table is easier to see as a distribution than as a dictionary. The following computation turns the counts into per-alias priors, then the figure compares the prior mass of the three ambiguous aliases.
PythonIn [6]Expand
alias_prior_rows = []
for alias, counts in ALIAS_COUNTS.items():
total = sum(counts.values())
for entity, count in counts.items():
alias_prior_rows.append(
{
"alias": alias,
"entity": entity,
"prior": count / total,
}
)
alias_prior_df = pd.DataFrame(alias_prior_rows)
ambiguous_aliases = ["mercury", "apple", "jordan"]
ambiguous_prior_df = alias_prior_df[
alias_prior_df["alias"].isin(ambiguous_aliases)
].copy()
ambiguous_prior_df["alias"] = pd.Categorical(
ambiguous_prior_df["alias"], categories=ambiguous_aliases, ordered=True
)
ambiguous_prior_df = ambiguous_prior_df.sort_values(
["alias", "prior"], ascending=[True, False]
).reset_index(drop=True)
ambiguous_prior_df["x"] = np.arange(len(ambiguous_prior_df))VisualizationOut [7]Expand
Now the test corpus and the gold labels. Each document has one or two sentences. Several contain distinguishing words. In d5, the toy annotation maps the individual apple to the fruit category in the KB. The wording in d8 can refer to the word Mercury itself. For this score comparison, we assign the planet as the gold label. After masking the matched alias, d8 leaves Mercury without description-word overlap, so we can isolate the graph contribution. We register gold labels as (doc_id, start, end, entity) so we can score predictions later.
PythonIn [8]Expand
DOCS = [
{
"doc_id": "d1",
"text": "Mercury is the closest planet to the Sun and completes an orbit every 88 days.",
},
{
"doc_id": "d2",
"text": "Mercury is a heavy silvery metal that is liquid at room temperature.",
},
{
"doc_id": "d3",
"text": "In Roman mythology Mercury was the messenger of the gods and the patron of commerce.",
},
{
"doc_id": "d4",
"text": "Apple reported record iPhone sales in its latest quarterly earnings.",
},
{
"doc_id": "d5",
"text": "She picked a ripe apple from the tree in the orchard.",
},
{
"doc_id": "d6",
"text": "Jordan signed with the Chicago Bulls in 1984 and won six championships.",
},
{
"doc_id": "d7",
"text": "Jordan is a country in Western Asia whose capital is Amman.",
},
{
"doc_id": "d8",
"text": "The Hubble article appears first. Mercury appears in the next paragraph.",
},
]
GOLD = [
("d1", 0, 7, "mercury_planet"),
("d2", 0, 7, "mercury_element"),
("d3", 19, 26, "mercury_god"),
("d4", 0, 5, "apple_inc"),
("d4", 22, 28, "iphone"),
("d5", 18, 23, "apple_fruit"),
("d6", 0, 6, "jordan_person"),
("d6", 23, 36, "chicago_bulls"),
("d7", 0, 6, "jordan_country"),
("d7", 53, 58, "amman"),
("d8", 4, 10, "hubble"),
("d8", 34, 41, "mercury_planet"),
]The gold records retain each mention's exact character span together with its entity ID. That detail matters because the evaluation will match on exact spans. A linker that picks the right entity but the wrong span will count as an error. This follows the strict span-and-entity convention described above. The toy corpus is also constructed so that each ambiguous alias appears in contexts with different evidence. That is what lets us compare the prior-only, local-only, and collective strategies.
Mention detection here is a gazetteer: match any alias as a whole word, case-insensitively. This detector has no learned span filter. A false detection on ordinary text would need mention rejection, separately from deciding whether an entity referent is absent from the KB.
PythonIn [9]Expand
ALIASES_SORTED = sorted(ALIAS_COUNTS, key=len, reverse=True)
ALIAS_BY_GROUP = {f"alias_{i}": alias for i, alias in enumerate(ALIASES_SORTED)}
ALIAS_PATTERN = re.compile(
r"\b(?:"
+ "|".join(
f"(?P<{group}>{re.escape(alias)})"
for group, alias in ALIAS_BY_GROUP.items()
)
+ r")\b",
re.IGNORECASE,
)
def detect_mentions(doc):
mentions = []
for match in ALIAS_PATTERN.finditer(doc["text"]):
alias = ALIAS_BY_GROUP[match.lastgroup]
mentions.append(
{
"doc_id": doc["doc_id"],
"surface": match.group(0),
"alias": alias,
"start": match.start(),
"end": match.end(),
"candidates": ALIAS_TO_CANDIDATES[alias],
}
)
return mentionsThe regular expression tries longer aliases first at a given position. If new york and new york city were both aliases, this ordering would choose the longer alternative when both are valid whole-word matches starting at the same character. This non-overlapping matcher does not enumerate nested mentions.
ConsoleOut [10]Expand
Documents: 8 Mentions detected: 12 Gold mentions: 12 d1 'Mercury' span=(0, 7) candidates=['mercury_element', 'mercury_planet', 'mercury_god'] d2 'Mercury' span=(0, 7) candidates=['mercury_element', 'mercury_planet', 'mercury_god'] d3 'Mercury' span=(19, 26) candidates=['mercury_element', 'mercury_planet', 'mercury_god'] d4 'Apple' span=(0, 5) candidates=['apple_inc', 'apple_fruit']
The matcher finds the twelve gold spans and no additional spans in this constructed corpus. That result depends on the chosen text and alias inventory. It does not establish how often the same matcher would miss entities or match nonentities elsewhere.
Next, the two scoring functions. Context similarity uses a binary-cosine over content words, which is a transparent stand-in for a TF-IDF or dense encoder. Stopwords come from scikit-learn's English list.
PythonIn [11]Expand
from sklearn.feature_extraction.text import ENGLISH_STOP_WORDS
def content_words(text):
tokens = re.findall(r"[a-z0-9]+", text.lower())
return {t for t in tokens if t not in ENGLISH_STOP_WORDS and len(t) > 1}
def cosine(a, b):
if not a or not b:
return 0.0
return len(a & b) / math.sqrt(len(a) * len(b))
ENTITY_CONTENT = {
eid: content_words(meta["desc"]) for eid, meta in ENTITIES.items()
}
def prior(entity, alias):
counts = ALIAS_COUNTS[alias]
return counts.get(entity, 0) / sum(counts.values())
def local_scores(mention, context, alpha=1.0, beta=10.0):
scores = {}
alias_words = content_words(mention["surface"])
context_evidence = context - alias_words
for eid in mention["candidates"]:
p = prior(eid, mention["alias"])
sim = cosine(context_evidence, ENTITY_CONTENT[eid] - alias_words)
scores[eid] = alpha * math.log(p) + beta * sim
return scoresThe filter removes listed stopwords and one-character tokens, then represents the remaining words as a set. cosine computes the cosine of binary word-presence vectors, returning zero when either set is empty. local_scores removes the surface words from both sides so the alias itself does not count as disambiguating evidence. The regex retains only ASCII letters and digits and does not lemmatize. Its English stopword list is not a multilingual tokenization policy, and scikit-learn documents limitations of that list. A changed similarity function needs its weights re-evaluated rather than assumed interchangeable.
ConsoleOut [12]Expand
mercury_planet local score = +4.782 mercury_element local score = -0.799 mercury_god local score = -1.897
In document d1, the context words planet, sun, and days overlap heavily with the planet description, and the context term with overwhelms the small prior advantage that the element has. With and , we multiply cosine similarity by ten before adding the log-prior term; this does not mean ten times as much evidential trust because the features have different scales. The result is a ranking score that puts the planet first for this mention. If all three candidate similarities were zero, these positive weights would leave the prior to select the element.
The same prior-plus-context score can be decomposed into its two contributions. The next computation isolates the log-prior term and the context-similarity term for the Mercury candidates in document d1.
PythonIn [13]Expand
mercury_demo = next(
m for m in detect_mentions(DOCS[0]) if m["alias"] == "mercury"
)
mercury_demo_context = content_words(DOCS[0]["text"])
mercury_demo_rows = []
for entity in mercury_demo["candidates"]:
p = prior(entity, "mercury")
sim = cosine(
mercury_demo_context - content_words(mercury_demo["surface"]),
ENTITY_CONTENT[entity] - content_words(mercury_demo["surface"]),
)
mercury_demo_rows.append(
{
"entity": entity,
"log_prior": math.log(p),
"context_term": 10.0 * sim,
"local_score": local_scores(mercury_demo, mercury_demo_context)[
entity
],
}
)
mercury_demo_df = (
pd.DataFrame(mercury_demo_rows)
.sort_values("local_score", ascending=False)
.reset_index(drop=True)
)
mercury_demo_df["x"] = np.arange(len(mercury_demo_df))VisualizationOut [14]Expand
Now the coherence graph. We model Wikipedia's inlink structure as a set of articles, each containing a set of entities. Milne-Witten relatedness is computed from these article sets, and we clamp the empty-intersection case to zero.
PythonIn [15]Expand
ARTICLES = {
"Solar_System": {"mercury_planet", "venus", "earth", "sun"},
"Mercury_(planet)": {"mercury_planet", "sun", "solar_system"},
"Mercury_(element)": {"mercury_element", "thermometer", "chemistry"},
"Roman_mythology": {"mercury_god", "jupiter_god", "hermes"},
"Hermes": {"hermes", "mercury_god", "greek_mythology"},
"Apple_Inc.": {"apple_inc", "iphone", "steve_jobs", "cupertino"},
"Apple": {"apple_fruit", "orchard", "vitamin_c"},
"Michael_Jordan": {"jordan_person", "chicago_bulls", "basketball", "nba"},
"Chicago_Bulls": {"chicago_bulls", "basketball", "nba", "jordan_person"},
"Jordan": {"jordan_country", "amman", "middle_east"},
"Amman": {"amman", "jordan_country"},
"iPhone": {"iphone", "apple_inc", "steve_jobs"},
"Hubble_Space_Telescope": {"hubble", "nasa", "telescope"},
"Great_Observatories": {"hubble", "mercury_planet", "telescope"},
}
ENTITY_ARTICLES = defaultdict(set)
for article, members in ARTICLES.items():
for member in members:
ENTITY_ARTICLES[member].add(article)
N_ARTICLES = len(ARTICLES)
def relatedness(a, b):
A, B = ENTITY_ARTICLES[a], ENTITY_ARTICLES[b]
if not A or not B:
return 0.0
shared = len(A & B)
if shared == 0:
return 0.0
numerator = math.log(max(len(A), len(B))) - math.log(shared)
denominator = math.log(N_ARTICLES) - math.log(min(len(A), len(B)))
return max(0.0, 1.0 - numerator / denominator) if denominator > 0 else 1.0ARTICLES is a hand-built incidence table, not a scraped Wikipedia graph. Its keys identify toy source articles; each value lists entity IDs linked from that article. Inverting it gives each entity's inlink set. In particular, the planet and Hubble share Great_Observatories, and Michael Jordan and the Bulls share two source articles. These chosen overlaps create the rewards we want to inspect.
ConsoleOut [16]Expand
relatedness(mercury_planet , hubble ) = 0.435 relatedness(mercury_element , hubble ) = 0.000 relatedness(jordan_person , chicago_bulls ) = 1.000 relatedness(jordan_country , chicago_bulls ) = 0.000
jordan_person shares two articles with chicago_bulls, so their relatedness is positive, while the country shares nothing and scores zero. mercury_planet shares the Great_Observatories article with hubble, providing a bonus we will inspect in d8. These values are compatibility scores for this graph, not calibrated probabilities of a semantic relation. A value of 0.6 does not mean a 60 percent chance that the entities are related.
Now the collective objective. The tiny candidate products in these documents let us enumerate every assignment and compare their computed floating-point scores.
PythonIn [17]Expand
GAMMA = 1.0
def collective_assignment(tables, candidates_per_mention, gamma=GAMMA):
best_combo, best_total = None, -np.inf
for combo in product(*candidates_per_mention):
total = sum(tables[i][combo[i]] for i in range(len(combo)))
for i in range(len(combo)):
for j in range(i + 1, len(combo)):
total += gamma * relatedness(combo[i], combo[j])
if total > best_total:
best_total, best_combo = total, combo
return best_combo, best_totalEnumeration cost grows with the product of the candidate-set sizes. With three candidates per mention and five mentions, the Cartesian product has 243 combinations. With ten mentions, it has 59,049. With twenty, it has over 3 billion. When enumeration exceeds the budget, use a different solver or an approximation with stated stopping rules. The toy enumerates all assignments, so it has neither sampling error nor an iterative convergence criterion.
PythonIn [18]Expand
rows = []
for doc in DOCS:
mentions = detect_mentions(doc)
context = content_words(doc["text"])
tables = [local_scores(m, context) for m in mentions]
mfs = [
max(m["candidates"], key=lambda e: prior(e, m["alias"]))
for m in mentions
]
loc = [max(t, key=t.get) for t in tables]
coll, _ = collective_assignment(tables, [m["candidates"] for m in mentions])
for i, m in enumerate(mentions):
rows.append(
{
"doc": m["doc_id"],
"surface": m["surface"],
"start": m["start"],
"end": m["end"],
"most_frequent": mfs[i],
"local_only": loc[i],
"collective": coll[i],
"local_score": round(tables[i][loc[i]], 3),
}
)
results = pd.DataFrame(rows)| doc | surface | start | end | most_frequent | local_only | collective | local_score | |
|---|---|---|---|---|---|---|---|---|
| 0 | d1 | Mercury | 0 | 7 | mercury_element | mercury_planet | mercury_planet | 4.782 |
| 1 | d2 | Mercury | 0 | 7 | mercury_element | mercury_element | mercury_element | 5.526 |
| 2 | d3 | Mercury | 19 | 26 | mercury_element | mercury_god | mercury_god | 4.649 |
| 3 | d4 | Apple | 0 | 5 | apple_inc | apple_inc | apple_inc | 0.545 |
| 4 | d4 | iPhone | 22 | 28 | iphone | iphone | iphone | 1.336 |
| 5 | d5 | apple | 18 | 23 | apple_inc | apple_fruit | apple_fruit | 1.450 |
| 6 | d6 | Jordan | 0 | 6 | jordan_country | jordan_person | jordan_person | 3.160 |
| 7 | d6 | Chicago Bulls | 23 | 36 | chicago_bulls | chicago_bulls | chicago_bulls | 0.000 |
| 8 | d7 | Jordan | 0 | 6 | jordan_country | jordan_country | jordan_country | 6.098 |
| 9 | d7 | Amman | 53 | 58 | amman | amman | amman | 3.381 |
| 10 | d8 | Hubble | 4 | 10 | hubble | hubble | hubble | 0.000 |
| 11 | d8 | Mercury | 34 | 41 | mercury_element | mercury_element | mercury_planet | -0.799 |
The table compares three strategies on every mention. The most_frequent column ignores context and always picks the prior argmax. The local_only column uses the prior plus context similarity for each mention independently. The collective column adds the coherence term and optimizes the whole document jointly. The local_score column reports the score of the local_only choice. The table shows where the strategies diverge. The component breakdowns quantify the individual score terms.
Now score all three strategies against the gold labels. A prediction earns credit only when its span and entity ID match the gold annotation.
PythonIn [20]Expand
gold_by_span = {(doc_id, start, end): eid for doc_id, start, end, eid in GOLD}
def accuracy(column):
correct = 0
for _, row in results.iterrows():
span = (row["doc"], row["start"], row["end"])
if gold_by_span.get(span) == row[column]:
correct += 1
return correct / len(results)
scores = {
col: accuracy(col) for col in ["most_frequent", "local_only", "collective"]
}
print(f"gold mentions: {len(GOLD)} predicted mentions: {len(results)}")
for name, value in scores.items():
print(f" {name:14s} accuracy = {value:.3f}")For each strategy, the function divides the number of correct span-and-entity predictions by the number of prediction rows. An empty results frame would make it divide by zero. Here each of the twelve unique gold spans has exactly one prediction row, with no extra rows, and each strategy predicts a KB ID. Under those conditions the fraction is accuracy over the fixed detected mentions. It is not a general end-to-end recall or F1 calculation: with missed gold spans, its denominator would not count those misses. Repeated correct rows would each get credit, even if another gold span was missed. For another corpus, compute detection recall and candidate recall separately, and use one-to-one matches between predictions and gold annotations for the specified linking precision and recall.
The most-frequent-sense baseline is the weakest by a wide margin: it only wins where the dominant sense happens to be correct. Local scoring fixes most of the errors by reading the context. Collective scoring fixes the remaining one.
On these twelve spans, local scoring changes four prior-only decisions. Adding coherence changes one more, so the collective and prior-only strategies differ on five spans. The next computation turns the accuracy scores into a plotting table, and the figure shows the gap.
PythonIn [21]Expand
accuracy_plot_df = pd.DataFrame(
{
"strategy": ["most_frequent", "local_only", "collective"],
"accuracy": [
scores["most_frequent"],
scores["local_only"],
scores["collective"],
],
}
)VisualizationOut [22]Expand
That remaining case is d8. Each toy document has exactly one mention with multiple candidates. With every other choice fixed, the pair rewards can only rescore that mention’s candidates. In d8, Hubble has one candidate, so the compatibility reward changes Mercury’s scores while Hubble’s assignment stays fixed.
PythonIn [23]Expand
d8 = next(d for d in DOCS if d["doc_id"] == "d8")
d8_mentions = detect_mentions(d8)
d8_context = content_words(d8["text"])
mercury_m = next(m for m in d8_mentions if m["alias"] == "mercury")
d8_local = local_scores(mercury_m, d8_context)
d8_combined = {
e: s + GAMMA * relatedness(e, "hubble") for e, s in d8_local.items()
}
d8_table = pd.DataFrame(
[
{
"entity": e,
"local": round(d8_local[e], 3),
"coherence_bonus": round(GAMMA * relatedness(e, "hubble"), 3),
"combined": round(d8_combined[e], 3),
}
for e in sorted(d8_local, key=lambda k: -d8_combined[k])
]
)| entity | local | coherence_bonus | combined | |
|---|---|---|---|---|
| 0 | mercury_planet | -0.916 | 0.435 | -0.481 |
| 1 | mercury_element | -0.799 | 0.000 | -0.799 |
| 2 | mercury_god | -1.897 | 0.000 | -1.897 |
In d8, removing mercury leaves no content-word overlap with any of its three descriptions. The local score therefore follows the prior and prefers the element. The planet's shared toy article with Hubble adds enough coherence reward to reverse that ranking. There was no local-score tie. We supplied the graph connection as additional evidence, and it can introduce errors of its own. Here the combined score matches the assigned planet label. The sentence alone leaves open whether Mercury refers to an entity.
Let's visualize the relatedness submatrix for nine selected entities from the collective objective.
PythonIn [25]Expand
HEAT_ENTITIES = [
"mercury_planet",
"mercury_element",
"mercury_god",
"hubble",
"apple_inc",
"apple_fruit",
"jordan_country",
"jordan_person",
"chicago_bulls",
]
REL_MATRIX = np.array(
[[relatedness(a, b) for b in HEAT_ENTITIES] for a in HEAT_ENTITIES]
)VisualizationOut [27]Expand
The off-diagonal cells connect the planet with Hubble and Michael Jordan with the Chicago Bulls. The Apple entities have no off-diagonal connection among the entities shown here; the company's iPhone neighbor is omitted from this matrix. The matrix is symmetric because swapping the two inlink sets leaves the score unchanged. These graph connections can supplement weak local evidence, but a document need not belong to a single topic. Most off-diagonal cells in this submatrix are zero. Coherence can change the selected assignment only when its weighted bonus changes the ordering of feasible assignments, including ties.
For each Mercury candidate in d8, the bars compare its local score with its score after adding compatibility with Hubble.
VisualizationOut [28]Expand
The coherence bonus changes the local ranking here by overcoming the prior gap. With a larger , a compatibility reward can also outweigh correct local evidence. The bar chart makes the tradeoff visible. The local score alone favors the element. The combined score favors the planet. The planet gains the pair reward derived from sharing the toy Great_Observatories inlink with Hubble.
Before accepting an ID, we can require the top two local scores to be separated by a margin. The following policy abstains on an empty candidate table or a margin below 0.2. A single candidate passes this margin check without an ambiguity comparison; it still needs separate checks for mention validity and KB coverage. The threshold is chosen for demonstration, not calibrated on validation data.
PythonIn [29]Expand
def choose_or_abstain(table, minimum_margin=0.2):
ranked = sorted(table, key=table.get, reverse=True)
if not ranked:
return "ABSTAIN"
if len(ranked) > 1 and table[ranked[0]] - table[ranked[1]] < minimum_margin:
return "ABSTAIN"
return ranked[0]
print("d1 local:", choose_or_abstain(demo_scores))
print("d8 local:", choose_or_abstain(d8_local))
print("empty candidates:", choose_or_abstain({}))ConsoleOut [29]Expand
d1 local: mercury_planet d8 local: ABSTAIN empty candidates: ABSTAIN
The policy accepts the planet for d1 and abstains on d8 and the empty table. Mercury's authored gold label in d8 is the planet, which is in the toy KB. The margin policy abstains on this in-KB gold target. An empty lookup also cannot establish KB absence. We have not implemented an open-world NIL detector or changed the three-strategy comparison above.
Comments
Add to the conversation
A question, a correction, or an example from your own work.
No comments yet. Be the first to share your thoughts!