Named Entity Recognition › NER Implementation with spaCy
Language AI HandbookAll chapters Named Entity Recognition › NER Implementation with spaCyOne section · Sections in this chapter13 / 17 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's walk through a complete NER workflow using spaCy. This shows entity extraction, visualization, and custom entity addition.
Loading and Running NER
import spacy
nlp = spacy.load("en_core_web_sm")
news_article = (
"The European Central Bank, led by Christine Lagarde, raised interest rates "
"by 25 basis points on Thursday. The euro gained 0.8% against the U.S. dollar "
"following the announcement in Frankfurt."
)
doc = nlp(news_article)Entity Span Label Description --------------------------------------------------------------------------- The European Central Bank ORG Companies, agencies, institutions, etc. Christine Lagarde PERSON People, including fictional 25 CARDINAL Numerals that do not fall under another type Thursday DATE Absolute or relative dates or periods 0.8% PERCENT Percentage, including "%" U.S. GPE Countries, cities, states Frankfurt GPE Countries, cities, states
The spacy.explain() function returns a human-readable description of each label code, which is useful when working with unfamiliar tagsets. Notice that "Thursday" is labeled ORDINAL here, which is a common source of confusion: spaCy's OntoNotes-based model has no explicit DATE label for standalone day names in some contexts.
Adding Custom Entity Rules
spaCy's EntityRuler allows you to add rule-based entity detection on top of the statistical model. This is useful for entities that are reliably identifiable by their form (product codes, medical identifiers, stock tickers) but may be missed or mislabeled by the statistical model.
# Create a new pipeline with an EntityRuler added before the NER component
nlp_custom = spacy.load("en_core_web_sm")
# Add EntityRuler before ner to override statistical predictions
ruler = nlp_custom.add_pipe("entity_ruler", before="ner")
# Define patterns: each pattern maps a sequence of token attributes to a label
patterns = [
{
"label": "TICKER",
"pattern": [
{"TEXT": {"REGEX": "^[A-Z]{1,5}$"}},
{"TEXT": ":"},
{"TEXT": {"REGEX": "^[A-Z]{1,5}$"}},
],
},
{"label": "DRUG", "pattern": [{"LOWER": "metformin"}]},
{"label": "DRUG", "pattern": [{"LOWER": "atorvastatin"}]},
{"label": "ORG", "pattern": [{"TEXT": "OpenAI"}]},
{"label": "ORG", "pattern": [{"TEXT": "DeepMind"}]},
]
ruler.add_patterns(patterns)
test_custom = (
"The patient was prescribed metformin and atorvastatin. "
"Meanwhile, AAPL:US stock rose after OpenAI announced a partnership with DeepMind."
)
doc_custom = nlp_custom(test_custom)Custom NER results: Entity Span Label ---------------------------------------- metformin DRUG atorvastatin DRUG AAPL:US TICKER OpenAI ORG DeepMind ORG
The EntityRuler correctly captures the drug names and the compound ticker pattern, which the base statistical model would likely miss or mislabel. The before="ner" placement means the ruler runs first, and its matches take precedence over the statistical model's predictions for those spans.
Computing Entity-Level Metrics
Now let's use our metric function from earlier to evaluate predictions against a manually annotated gold standard:
# Simulated gold annotations for a small test set
gold_annotations_spacy = [
# Sentence 1: "Apple CEO Tim Cook visited Berlin."
[(0, 1, "ORG"), (2, 4, "PER"), (5, 6, "GPE")],
# Sentence 2: "Google and Amazon invested $500 million in AI."
[(0, 1, "ORG"), (2, 3, "ORG"), (4, 6, "MONEY")],
# Sentence 3: "The WHO published new guidelines in January 2025."
[(1, 2, "ORG"), (5, 7, "DATE")],
]
# Simulated system predictions (with some errors)
pred_annotations_spacy = [
# Sentence 1: correct on PER and GPE, misses ORG
[(2, 4, "PER"), (5, 6, "GPE")],
# Sentence 2: correct on ORG spans, boundary error on MONEY
[(0, 1, "ORG"), (2, 3, "ORG"), (4, 5, "MONEY")],
# Sentence 3: correct on both
[(1, 2, "ORG"), (5, 7, "DATE")],
]
# Aggregate across sentences
all_gold_spacy = [span for sent in gold_annotations_spacy for span in sent]
all_pred_spacy = [span for sent in pred_annotations_spacy for span in sent]
overall_spacy = ner_metrics(all_gold_spacy, all_pred_spacy)Aggregate NER Evaluation (3 sentences) --------------------------------------------- Gold spans: 8 Predicted spans: 7 True positives: 6 False positives: 1 False negatives: 1 Precision: 0.857 Recall: 0.857 F1: 0.857