Named Entity Recognition › Gazetteers and External Knowledge
Language AI HandbookAll chapters Named Entity Recognition › Gazetteers and External KnowledgeOne section · Sections in this chapter12 / 17 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
While transformer models have largely replaced hand-crafted features, gazetteers and external knowledge sources remain valuable as augmentation, particularly in specialized domains.
A gazetteer is a curated list of entity mentions associated with entity types. For example:
- A gazetteer of company names from a financial database, updated daily
- A gazetteer of drug names from PubChem or DrugBank
- A gazetteer of geographic names from GeoNames
Gazetteers can be used in several ways. The simplest approach is post-processing: after a model makes predictions, override low-confidence predictions if the token matches a gazetteer entry with high reliability. A more principled approach uses gazetteer membership as an additional input feature, either by appending an embedding derived from the gazetteer lookup to the token representation, or by adding a Boolean feature indicating gazetteer match.
# Simple gazetteer-augmented post-processing
gazetteers = {
"ORG": {
"Apple",
"Google",
"Microsoft",
"Amazon",
"Meta",
"Tesla",
"OpenAI",
},
"PER": {"Biden", "Obama", "Trump", "Musk", "Zuckerberg", "Cook", "Altman"},
"LOC": {
"Paris",
"London",
"Berlin",
"Tokyo",
"Beijing",
"Washington",
"California",
},
"GPE": {"USA", "UK", "Germany", "France", "China", "Japan", "India"},
}
# Reverse lookup: word -> most likely type
word_to_type = {}
for entity_type, words in gazetteers.items():
for word in words:
word_to_type[word] = entity_type
def apply_gazetteer_override(
predictions, word_to_type_map, confidence_threshold=0.8
):
"""
Override low-confidence predictions with gazetteer knowledge.
Only overrides if the model's confidence is below threshold AND
the word appears in the gazetteer.
"""
corrected = []
for pred in predictions:
word = pred["word"].strip()
if pred["score"] < confidence_threshold and word in word_to_type_map:
original_type = pred["entity_group"]
corrected_type = word_to_type_map[word]
corrected.append(
{
**pred,
"entity_group": corrected_type,
"overridden": True,
"original_type": original_type,
}
)
else:
corrected.append({**pred, "overridden": False})
return corrected
# Test on a sentence with potentially ambiguous entity
test_text_gaz = (
"Cook announced that Apple would partner with OpenAI in Cupertino."
)
raw_preds = ner_pipeline(test_text_gaz)
corrected_preds = apply_gazetteer_override(raw_preds, word_to_type)Gazetteer-augmented predictions: Entity Type Score Status ------------------------------------------------------- Cook PER 0.997 model prediction Apple ORG 0.999 model prediction OpenAI ORG 0.995 model prediction Cupertino LOC 0.998 model prediction
Gazetteers are most valuable when the entity vocabulary is well-defined and stable, and when the model encounters terms that are underrepresented in its pre-training data. For general-purpose NER on standard newswire text, gazetteers add little beyond what a well-trained transformer already knows. In specialized domains like pharmaceutical NER, where every approved drug name must be correctly extracted, gazetteer augmentation is often essential.