Information Extraction › Named Entity Recognition

Named entity recognition is the entry point for nearly all IE pipelines. Before you can extract a relationship like "Apple acquired Beats," you need to know that "Apple" is an organization and "Beats" is another organization. NER is what provides that grounding: a systematic scan of text that finds and labels every entity mention.

The term "named entity" was introduced in the MUC-6 evaluation in 1996 and originally referred to proper names of specific real-world entities: people, places, organizations. Modern usage has expanded to include temporal expressions, monetary values, percentages, and other categorically meaningful mentions that lack proper names. This expansion reflects practical needs: a financial extraction system needs to identify "$3 billion" as a money amount just as much as it needs to identify "Apple" as an organization.

Why does NER matter so much as a first step? Because almost every downstream IE task depends on knowing what kind of thing each mention refers to. Relation extraction uses entity types to filter candidate pairs: you check for employment relations between PERSON-ORG pairs, not between DATE-MONEY pairs. Event extraction uses entity types to assign argument roles: a PERSON entity near a firing verb is more likely the employee than the organization. Without NER, you are working with anonymous strings, and the combinatorial explosion of possible pairings makes precise extraction nearly impossible at scale.

An NER system scans a piece of text and labels each entity mention with a type. The standard entity categories, established by datasets like CoNLL-2003 and OntoNotes, include:

• PER (Person): "Barack Obama," "Marie Curie," "Elon Musk"
• ORG (Organization): "Apple," "NASA," "European Union"
• LOC (Location): "Mount Everest," "the Amazon River"
• GPE (Geopolitical Entity): "France," "New York City," "the European Union" (when used as a political entity)
• DATE / TIME: "March 15," "9:30 AM," "the 1990s"
• MONEY: "$3 billion," "€200 million"
• PRODUCT: "iPhone 15," "Model 3"

Different frameworks use slightly different taxonomies. spaCy's en_core_web_trf model recognizes 18 entity types following OntoNotes conventions. BERT-based biomedical models may add GENE, DISEASE, and CHEMICAL. The right taxonomy depends on your domain.

The boundary between types is not always clean. "Washington" can be a person (George Washington), a place (Washington state), or a geopolitical entity (Washington D.C.). "March" is usually a month, but in the sentence "The soldiers were ordered to march" it is a verb, not an entity at all. Context resolves these ambiguities, and this is precisely why modern contextual encoders like BERT outperform earlier feature-engineered systems. A static dictionary lookup for "Washington" cannot know which sense applies; a transformer that has seen thousands of sentences with "Washington D.C." and "Washington state" can infer the correct type from the surrounding tokens.

NER is typically framed as a sequence labeling problem. Given a sequence of $n$ tokens $\mathbf{x} = (x_1, x_2, \ldots, x_n)$, the model assigns one tag $y_i$ to each token $x_i$. The most common tagging scheme is BIO (also called IOB2), where each tag belongs to the set:

$$y_i \in \{\text{B-TYPE}, \text{I-TYPE}, \text{O}\}$$

where:

• $\text{B-TYPE}$ marks the Beginning of an entity of the given type (e.g., B-PER for the start of a person name)
• $\text{I-TYPE}$ marks a token Inside (continuing) an entity of the same type (e.g., I-PER for a second or later token of a person name)
• $\text{O}$ marks a token Outside any entity

For the sentence "Marie Curie won the Nobel Prize":

The BIO scheme lets spans of arbitrary length be represented with only a small tag vocabulary. A system that predicts these tags can locate and type any entity, regardless of how many tokens it covers. Multi-token entities are identified by reading consecutive B and I tags of the same type. The B marker is critical: it distinguishes between a single entity "New York" (B-GPE, I-GPE) and two separate entities "New" and "York" appearing back to back (B-GPE, B-GPE).

Modern transformer-based NER models compute a contextual representation $\mathbf{h}_i$ for each token $x_i$ via the encoder, then pass it through a linear classification head:

$$P(y_i \mid \mathbf{x}) = \text{softmax}(\mathbf{W} \mathbf{h}_i + \mathbf{b})$$

where:

• $\mathbf{h}_i \in \mathbb{R}^d$ is the contextual token embedding produced by the transformer encoder for position $i$
• $\mathbf{W} \in \mathbb{R}^{|\mathcal{Y}| \times d}$ is a learned weight matrix mapping from embedding space to tag scores
• $\mathbf{b} \in \mathbb{R}^{|\mathcal{Y}|}$ is a learned bias vector
• $|\mathcal{Y}|$ is the number of possible tags (e.g., $2 \times |\text{types}| + 1$ for BIO)

Because the encoder sees the full sentence bidirectionally, it can resolve ambiguities that left-to-right models would miss: "Washington" near "D.C." is a GPE, while "Washington" near "the Founding Fathers" is a PER. This bidirectional context was the primary advantage that BERT-based NER systems demonstrated over LSTM-CRF systems when they were introduced in 2018 to 2019.

Some systems add a conditional random field (CRF) layer on top of the per-token predictions. The CRF models tag transition probabilities explicitly, making sure that the output sequence is globally consistent: for example, an I-PER tag cannot follow a B-ORG tag because that would imply the same token continues two different entities. In practice, modern large transformers are expressive enough that the CRF layer provides only marginal improvement, but it can still help on smaller datasets or when the training set has inconsistent annotations.

The following visualization shows the typical NER F1 performance of different system classes on the CoNLL-2003 English benchmark, illustrating how accuracy has improved from rule-based systems through statistical models to transformer-based approaches.

spaCy is the most widely used NLP library for production NER. Its transformer-backed en_core_web_trf model achieves state-of-the-art accuracy on standard benchmarks with minimal setup. For this walkthrough we use the smaller en_core_web_sm model, which is faster and sufficient to illustrate the key concepts.

The model identifies "Apple" and "Beats Electronics" as organizations, "$3 billion" as a money amount, "May 2014" as a date, and "Cupertino, California" as a geopolitical entity. Notice that "Tim Cook" is recognized as a person, and the character offsets tell you exactly where in the source text each entity was found. These offsets are valuable for downstream tasks: you can display entities highlighted in the original text, or you can use them to extract the surrounding context for relation extraction.

One subtlety to notice: "Apple's" is correctly resolved to "Apple" as an organization, even though the possessive suffix "'s" is attached. spaCy's tokenizer splits the "'s" into a separate token, and the NER model correctly assigns the entity span to just "Apple". This kind of tokenization-aware entity boundary detection is straightforward for well-resourced languages like English but becomes trickier for languages with rich morphology, where a single word form may encode the entity plus several grammatical markers.

Standard BIO labeling cannot represent nested entities: in the phrase "University of California, Berkeley," both "University of California, Berkeley" and "California" are valid entities, but a flat BIO tagger can only assign each token to one span. More complex schemes like BIOUL (Begin, Inside, Outside, Unit, Last) or nested NER architectures handle this, but for most practical pipelines, flat NER is sufficient. The convention is to label the longest containing span.

The nested entity problem is more common in domains like biomedicine, where gene names, protein complexes, and cell types frequently appear as constituents of longer named phrases. A gene name like "TP53" may appear inside a phrase like "TP53 mutation," which is itself a clinical finding. Dedicated nested NER architectures enumerate all possible spans and classify each one independently, treating the problem as multi-label span classification rather than a sequential tagging problem. This increases computational cost but allows complete annotation of overlapping structures.

A related challenge is entity mention disambiguation. The text "Apple" can refer to the technology company, the fruit, or a music label. NER labels the mention's type but does not link it to a unique knowledge base entry. That step, called entity linking or entity disambiguation, pairs NER with a lookup against a knowledge base like Wikidata or Freebase to produce canonical identifiers. Systems like REL and BLINK train dedicated entity linking models that score candidate knowledge base entries against the surrounding context and return the best match. Entity linking is a prerequisite for knowledge graph population, where the same entity should always map to the same node regardless of how it is mentioned in text. Without it, a system may create duplicate nodes for "Apple Inc.," "Apple," and "AAPL" that should all point to the same company.