Named Entity Recognition: Types, BIO Tagging & Evaluation

Text is full of references to real-world things: people, companies, countries, dates, dollar amounts. Named Entity Recognition (NER) is the task of automatically identifying these mentions in raw text and classifying them into predefined categories such as person, organization, or location.

Consider the sentence: "Elon Musk announced that Tesla will open a new factory in Germany by March 2026." A NER system should identify "Elon Musk" as a person, "Tesla" as an organization, "Germany" as a location, and "March 2026" as a date. This extracted structure powers an enormous range of downstream applications: search engines that understand queries about specific people, question-answering systems that can retrieve factual information, business intelligence pipelines that monitor mentions of competitors, and biomedical systems that extract drug-disease relationships from clinical notes.

NER combines sequence labeling with information extraction. Like Part-of-Speech (POS) tagging (covered in the previous chapter), NER assigns labels to tokens. But NER introduces a harder challenge: entities span multiple tokens, and detecting where an entity begins and ends is itself a non-trivial problem. This chapter covers the entity type taxonomy, how NER is formulated as a sequence labeling problem, the specific difficulty of boundary detection and nested entities, the long evolution of NER architectures from hand-written rules through CRFs to transformer-based models, and the major datasets and benchmarks that have shaped the field.

NER systems classify entities into a predefined taxonomy of types. The categories chosen depend on the application domain, but a small set of core types recurs across virtually every NER benchmark and production system.

The three foundational categories that appear in nearly every NER taxonomy are:

• PER (Person): Names of individual people, real or fictional. Examples include "Marie Curie", "Barack Obama", and "Sherlock Holmes". Titles attached to names ("Dr. Fauci", "President Lincoln") are typically included as part of the span.
• ORG (Organization): Companies, government agencies, universities, sports teams, and other collective entities. Examples include "NASA", "Google", "Harvard University", and "the World Health Organization".
• LOC (Location): Places, including countries, cities, regions, rivers, mountains, and physical addresses. Examples include "New York City", "the Amazon River", and "Mount Everest". Some tagsets split this further into GPE (geopolitical entities, like countries and cities) and LOC (pure geographic features, like mountains and rivers).

Extended taxonomies add categories for specific applications:

• DATE / TIME: Calendar dates and time expressions. Examples include "January 2024", "last Tuesday", and "3:00 PM EST".
• MONEY: Monetary amounts. Examples include "$50 million", "€200 billion", and "two hundred dollars".
• PERCENT: Percentage expressions. Examples include "7.5%", "a third", and "half".
• MISC (Miscellaneous): Entities that do not fit the above categories but are still proper nouns: nationality adjectives ("French", "Japanese"), product names, events, and works of art.

General-purpose tagsets are designed for broad applicability, but specialized domains benefit from finer-grained categories. Biomedical NER systems distinguish GENE, PROTEIN, DISEASE, CHEMICAL, SPECIES, and CELL_LINE. Legal NER systems label STATUTE, COURT, JUDGE, PLAINTIFF, and DEFENDANT. Financial NER systems need TICKER_SYMBOL, EXCHANGE, and FISCAL_PERIOD. These domain-specific systems are trained on annotated corpora from the target domain, because the distributional properties of entity mentions differ substantially from newswire text.

The choice of taxonomy is not purely a technical decision. It reflects assumptions about what information consumers of the extracted data need. A system designed to populate a corporate knowledge graph cares deeply about the distinction between a subsidiary (ORG) and its parent company (also ORG) in a way that a general news reader does not. Before designing or selecting a NER system for an application, it is worth asking: which entity types matter for downstream tasks, and at what granularity?

Let's load spaCy and inspect what entity types its general-purpose English model recognizes:

The output shows each entity span, its type label, and its character offsets in the source string. spaCy's en_core_web_sm model uses an extended tagset that includes GPE (geo-political entity) as a distinct category from LOC. This reflects the CoNLL and OntoNotes annotation traditions. The character offsets are useful for highlighting entities in a UI or aligning predictions to the original text.

Different domains contain different mixes of entity types. Understanding the expected distribution helps when choosing a pre-trained model or designing a training corpus.

The chart confirms the intuition: financial text is dominated by MONEY and PERCENT entities, biomedical text mixes ORG and PERSON with product references, and news text skews heavily toward people and places. A model trained only on newswire will generalize poorly to financial filings, which is why domain adaptation or domain-specific training is important in practice.

The cleanest formulation of NER turns it into a token-level classification problem. Rather than predicting entity spans directly, we assign a label to each token and use a tagging scheme that encodes both the entity type and the position of the token within a multi-token span.

This framing has a powerful practical consequence: any sequence labeling model, from a simple logistic regression to a transformer, can be applied to NER without fundamental changes to the model architecture. The NER-specific engineering lives in the tagging scheme and the loss function, not in the model itself.

The most widely used scheme is BIO (also called IOB2). Each token receives one of three tag prefixes:

• B-TYPE: The token begins an entity of type TYPE.
• I-TYPE: The token is inside a continuing entity of type TYPE.
• O: The token is outside any entity.

For the sentence "Tim Cook visited Paris", the BIO labels would be:

This encoding turns span detection into token classification. A model that correctly predicts B-PER then I-PER for adjacent tokens implicitly identifies the span "Tim Cook" as a person entity. Span recovery from BIO tags is straightforward: a new entity begins whenever a B- tag appears or an I- tag follows a different entity type.

The BIO tagging scheme and its variants (BIOES, BILOU) are covered in detail in the next chapter on BIO Tagging. For now, the key insight is that this encoding makes NER amenable to any sequence labeling model: HMMs, CRFs, or neural architectures with token-level classification heads.

Given BIO predictions, recovering entity spans requires a simple decoding pass:

The decoder correctly recovers "Tim Cook" as a single PER span (tokens 0-2), "Paris" as LOC (token 3-4), and "London" as LOC (token 5-6). The function also handles the case where an I- tag follows a different entity type (a common tagging inconsistency), treating it as a new begin rather than crashing.

Identifying where entities begin and end is arguably harder than classifying entity types. Boundary errors account for a large fraction of NER mistakes in practice.

Several linguistic phenomena make boundary detection difficult:

Prepositional attachment. Consider "the University of Michigan professor." Is the entity "University of Michigan" (a named institution) or "University of Michigan professor" (not an entity)? The correct boundary depends on semantic interpretation, not surface syntax.

Possessives and determiners. "Apple's quarterly earnings" contains "Apple" as an ORG entity. Should the possessive "'s" be included? Convention differs across annotation schemes. Most English NER corpora exclude the possessive clitic from the entity span, but this is a choice, not a linguistic necessity.

Titles and honorifics. "President Biden" versus "Biden": is the title part of the entity? The CoNLL 2003 annotation guidelines treat "President Biden" as a single PER span, while some other guidelines treat "Biden" alone as the entity.

Coordination. "Amazon and Google both reported profits" contains two ORG entities, but the surface form shares no obvious boundary marker. Systems trained on isolated entity mentions sometimes fail to split coordinated entities correctly.

Abbreviations. "the U.S." and "the United States" refer to the same entity, but their spans look completely different to a tokenizer.

These ambiguities are not merely academic curiosities. They translate directly into inter-annotator disagreement during corpus construction, and that disagreement propagates into the training data. When two annotators would choose different boundaries for the same span, a model trained on their combined labels learns an inconsistent mapping and produces boundary errors at inference time. Understanding the linguistic sources of boundary ambiguity helps both in designing annotation guidelines and in diagnosing model failures.

One of the least-discussed but most important factors in NER system quality is the annotation guideline document. Every NER benchmark comes with a guide that specifies, for hundreds of edge cases, exactly which tokens should be included in an entity span and what type label to assign. The CoNLL 2003 annotation guidelines run to dozens of pages. The OntoNotes guidelines are even longer.

These decisions have real consequences. The guideline that says "include titles like President and Dr. in the PER span" means a model trained on CoNLL 2003 will learn to extend person spans to include honorifics, while a model trained on a dataset with a different guideline will learn the opposite behavior. When you use a pre-trained NER model on new data, you are implicitly adopting the annotation conventions of whatever corpus that model was trained on. Mismatches between training conventions and deployment conventions are a common, underappreciated source of NER errors in production.

One of the most challenging aspects of NER in real-world text is that entities can be nested: one entity span can contain another as a component.

Standard BIO tagging cannot represent nested entities: a token can have only one tag, so the inner entity must be silently dropped. The sentence "the New York Times" would typically be tagged as a single ORG span, even though "New York" is also a valid LOC entity.

Several approaches address nested NER:

• Layer-by-layer annotation: Annotate one layer of entities at a time, allowing multiple tag sequences per sentence. The ACE 2004 and 2005 datasets use this approach.
• Span-based models: Instead of token-level tagging, enumerate all possible spans and classify each independently. This naturally handles nesting at the cost of $O(n^2)$ candidate spans.
• Constituency parsing integration: Treat nested entities as a variant of the phrase structure parsing problem.
• Graph-based representations: Encode entities as nodes in a graph, with nesting represented by containment edges.

For most practical applications, flat NER (ignoring nesting) is sufficient. Nested NER matters in biomedical information extraction, where phrases like "human immunodeficiency virus type 1 protease inhibitor" contain multiple entity spans of different types.

Let's visualize how nested entities arise:

The flat annotation captures "The New York Times" as a single ORG entity, while the nested annotation additionally captures "New York" as a GPE entity contained within it. Standard BIO tagging cannot represent both annotations simultaneously for the same token sequence.

NER has a rich history spanning four decades and three major paradigm shifts. Understanding this history clarifies why modern systems are designed the way they are and what problems each generation of approaches was solving.

The earliest NER systems were built from hand-written rules and gazetteers. A gazetteer is a list of known entity names, for instance a list of country names, a list of major corporations, or a list of common first and last names. Rules then check whether a token appears in one of these lists, or whether contextual patterns like "Mr. [WORD]" or "[WORD] Inc." indicate an entity.

Rule-based systems have real advantages: they are interpretable, they never hallucinate, and they can be updated by simply editing the rules or adding entries to gazetteers. They remain useful in domains where the entity vocabulary is small and stable, such as a financial system that only needs to recognize a fixed list of stock tickers. Early systems like Proper Noun Analyzer (PROLEX) and FASTUS (1995) achieved surprisingly good performance on newswire NER using these techniques.

Their fundamental weakness is coverage and maintainability. Languages are productive: new entities are created constantly (new companies, new public figures, new events), and hand-written rules cannot keep pace. Rules also interact in complex ways as the rule set grows, making maintenance error-prone. By the late 1990s, the community was actively exploring statistical alternatives.

The first statistical NER systems used Hidden Markov Models (HMMs). An HMM models the joint probability of a token sequence and its label sequence using emission probabilities (how likely is this word given this label?) and transition probabilities (how likely is this label given the previous label?). Training requires only annotated data, which removes the need for manual rule writing. We covered HMMs in detail in the previous chapter on sequence labeling.

HMMs have a significant limitation for NER: they rely on each token being independent given the current label. In practice, NER decisions depend on rich context: whether a word is capitalized, whether it is preceded by a title, whether it appears in a list of known organizations, and dozens of other features. Capturing these features within an HMM requires increasingly complex emission probability models.

Conditional Random Fields (CRFs), introduced by Lafferty, McCallum, and Pereira in 2001, addressed this limitation directly. A linear-chain CRF models the conditional distribution $P(\mathbf{y} | \mathbf{x})$ directly rather than the joint distribution, which means it can condition on arbitrary features of the entire input sequence without needing to model the input distribution. A typical CRF feature function for NER might fire when the current token is capitalized and the previous label was B-PER, assigning that configuration a learned weight.

The standard feature set for CRF-based NER includes:

• Word identity: the exact lowercased word form
• Word shape: capitalization pattern (Xxxx for Title Case, XXXX for all caps, xxxx for all lower)
• Prefix and suffix patterns: the first and last 2-4 characters
• Gazetteer membership: whether the word appears in a person/organization/location list
• Part-of-speech tag: the syntactic category
• Preceding and following words: context window of 1-2 tokens on each side
• Preceding and following labels: the predicted labels for surrounding tokens

CRFs with these features achieved state-of-the-art NER performance through most of the 2000s, reaching around 86-89 F1 on CoNLL 2003. The primary bottleneck was feature engineering: designing the right feature templates required substantial domain expertise and tuition through experimentation.

Let's demonstrate what a simple feature function for CRF-style NER looks like:

Each token generates a dictionary of binary and string-valued features. The CRF then learns a weight for each (feature, label) combination. At inference time, the Viterbi algorithm finds the label sequence with the highest total weighted score, taking into account both feature weights and label transition weights.

The shift from hand-engineered features to learned representations began with word embeddings. Instead of encoding "does this word appear in a gazetteer?", neural NER systems encode each word as a dense vector that captures its distributional meaning. This frees the model from the need for manual feature engineering and allows it to generalize to unseen words based on semantic similarity to known words.

The architecture that defined this era was the BiLSTM-CRF, introduced by Lample et al. (2016) and Chiu and Nichols (2016). The model has three components working together:

First, a character-level convolutional network or BiLSTM reads each word's characters and produces a character-level embedding. This component captures morphological patterns, such as recognizing that words ending in "-son" (Dickson, Jackson, Peterson) tend to be surnames, or that fully capitalized words are often abbreviations. This matters enormously for NER because named entities frequently contain unusual words that are out of vocabulary for word-level models.

Second, the character embedding is concatenated with a pre-trained word embedding (GloVe or fastText) to form a combined token representation. The combined representation captures both morphological patterns and semantic associations.

Third, a bidirectional LSTM processes the sequence of token representations, producing a context-sensitive encoding for each token. At each position, the BiLSTM combines information from the left context (how has the text built up to this point?) and the right context (what comes after this token?). This bidirectional view is essential for NER: recognizing "Times" as part of an organization name requires knowing it follows "New York", and recognizing "Bank" as part of "Bank of America" requires knowing what follows it.

The BiLSTM outputs feed into a linear layer that produces logit scores for each possible BIO tag. These scores feed into a CRF layer that applies the Viterbi algorithm during decoding. The CRF layer enforces label transition constraints, such as the rule that an I-PER tag cannot immediately follow an O tag, which improves output consistency compared to independent token-level classification.

BiLSTM-CRF systems achieved around 90-91 F1 on CoNLL 2003, an improvement of 4-5 points over classical CRFs. The gain came almost entirely from the learned representations replacing hand-engineered features. The core sequence labeling logic, including the CRF decoding, remained the same.

The introduction of BERT in 2018 created the current dominant paradigm for NER. Rather than training task-specific representations from scratch, BERT provides deeply contextual word representations pre-trained on large text corpora. Fine-tuning BERT for NER requires only adding a linear classification head on top of the token representations and training on labeled NER data.

The key advance over BiLSTM models is the quality of the representations. BERT uses self-attention (covered in detail in later chapters) to produce representations that integrate long-range context much more effectively than LSTMs. For NER, this means BERT can use cues from many positions away to determine entity type and boundaries. A sentence like "He said it was the Microsoft founder's decision" requires integrating information about the subject, the verb, and the noun phrase to correctly identify "Microsoft" as an ORG entity in that context.

With BERT-base fine-tuned on CoNLL 2003, NER performance jumped to around 92 F1. Larger models (BERT-large, RoBERTa-large) reached 93-94 F1. These numbers had seemed nearly impossible a few years earlier.

The transformer architecture also changed how practitioners think about NER for new domains. With classical CRFs or BiLSTM-CRFs, adapting to a new domain required substantial annotated training data to learn new feature weights or BiLSTM representations. With pre-trained transformers, the representations already encode rich world knowledge and linguistic patterns from large corpora. Fine-tuning on even a few hundred labeled examples in a new domain can produce a usable system, because the model is not learning representations from scratch.

Let's build a complete CRF-based NER pipeline to understand the mechanics. We will use the sklearn-crfsuite package, which provides a clean Python interface to CRF training.

The CRF correctly identifies "Andy Jassy" as a PER entity (relying on capitalization and the context of a preceding ORG), "Amazon" as an ORG, and "Berlin" as a LOC. Despite the extremely small training set, the model generalizes because the features (capitalization, POS tags, context words) generalize across entity types.

One of the most useful diagnostic tools for CRF models is inspecting the learned transition weights. High transition weights for B-PER to I-PER confirm the model learned that person names commonly have multiple tokens. High negative weights for O to I-PER confirm the model learned that an inside tag cannot appear without a preceding begin tag.

Measuring NER system performance requires care because there are multiple ways to define what counts as a correct prediction. Two distinct evaluation philosophies produce very different numbers for the same system.

In exact match evaluation (also called strict evaluation), a prediction is correct only if both the span boundaries and the entity type match the gold annotation exactly. A system that predicts "Tim Cook" as ORG instead of PER gets zero credit. A system that predicts "Cook" instead of "Tim Cook" also gets zero credit, even though it partially overlapped with the correct span.

Partial match evaluation (also called lenient evaluation) relaxes this requirement. It counts a prediction as correct if it overlaps with a gold span, even if the boundaries are imprecise or the type is wrong. Different variants of partial matching include:

• Type match: The entity type is correct, boundary is ignored.
• Partial boundary match: The predicted span overlaps with the gold span, regardless of type.
• Exact type + partial boundary: Type is correct and spans overlap.

NER evaluation uses precision, recall, and their harmonic mean F1, computed over entity spans rather than individual tokens.

Given a set of predicted entity spans $\hat{E}$ and gold spans $E^*$, the metrics are:

where:

• $\hat{E}$: the set of (span, type) pairs predicted by the system
• $E^*$: the set of (span, type) pairs in the gold annotation
• $|\hat{E} \cap E^*|$: the number of predicted spans that exactly match a gold span (in both boundary and type)

Precision measures how often the system's predictions are correct. Recall measures how often gold entities are found. F1 balances the two. A system that predicts fewer but more accurate entities will have high precision and low recall. A system that aggressively predicts many entities will have high recall and low precision.

NER systems typically report span-level F1 for each entity type separately (per-type F1) as well as a macro-averaged F1 across types and a micro-averaged F1 weighted by entity count.

Let's implement these metrics from scratch:

System A gets 2 true positives (the PER and LOC spans that exactly match gold), 1 false positive (the wrong-type ORG prediction), and 2 false negatives (the undetected DATE and the wrong-type ORG). System B also gets 2 true positives (PER and LOC), 1 false positive (the partial DATE span), and 2 false negatives (the exact DATE and the missing ORG). Both systems achieve an F1 of 0.57. This shows that type errors and boundary errors have symmetric penalties under exact match scoring.

This strictness is both a strength and a limitation of exact match evaluation. It creates an unambiguous ranking of systems, but it can understate the practical utility of a system that consistently finds entities with slightly off boundaries.

Boundary errors and type errors each account for roughly 10% of predictions in this simulation. Boundary errors are the more common failure mode in practice because they often arise from consistent annotation disagreements (whether to include titles, whether to extend a span to include possessives) rather than fundamental model failures.

Let's directly compare exact match and partial match evaluation on the same predictions to see how much the choice of evaluation scheme affects reported performance:

The partial match scores are higher because they award credit for the boundary-error MONEY span that exact match rejects. In practice, this gap can be substantial (5-15 points of F1) on domains where entity boundaries are ambiguous.

Several datasets have become standard reference points for NER system comparison. Understanding their origins, domains, and annotation conventions is essential for interpreting published results.

CoNLL 2003 is the most widely used English NER benchmark. It was released as part of the 2003 Conference on Computational Natural Language Learning shared task and consists of newswire text from the Reuters Corpus, annotated with four entity types: PER, ORG, LOC, and MISC.

The dataset contains approximately 22,000 sentences (roughly 300,000 tokens) split into training, development (testa), and test (testb) sets. The test set F1 score on this dataset is the standard comparison point for NER systems across a decade of research, enabling direct comparisons between systems published years apart.

A critical property of CoNLL 2003 is that its annotation guidelines are relatively conservative: abbreviations and ambiguous cases tend to be left unannotated rather than annotated with uncertain labels. This makes the dataset cleaner but also means systems tuned on it may not generalize well to noisier domains.

An important caveat about CoNLL 2003 scores: subsequent research has shown that the test set contains a significant number of annotation errors, and that near-human performance may already have been reached for this particular benchmark. Models reporting 94+ F1 on CoNLL 2003 may be partially benefiting from overfitting to specific annotation idiosyncrasies. This does not diminish the value of the dataset as a long-term comparison point, but it does suggest that further gains on this specific benchmark may not translate into equivalent gains on real-world NER tasks.

OntoNotes 5.0 is a larger and more linguistically diverse corpus covering newswire, broadcast news, broadcast conversation, web text, magazine articles, and telephone conversations. It uses an 18-class entity taxonomy (the same one underlying spaCy's default model), including CARDINAL, ORDINAL, QUANTITY, and LAW in addition to the standard PER/ORG/LOC/DATE/MONEY/PERCENT types.

Because OntoNotes includes spoken language transcripts and web text, it is harder and more representative of real-world variation than CoNLL 2003. State-of-the-art F1 on OntoNotes is roughly 5-8 points lower than on CoNLL 2003 for equivalent-sized models.

The Automatic Content Extraction (ACE) datasets were developed by NIST for the information extraction community. They annotate entities, relations, and events, making them useful for systems that need to extract structured knowledge rather than just entity mentions. ACE uses a nested entity annotation scheme, capturing both inner and outer entity spans.

The biomedical NLP community has developed specialized datasets for gene/protein NER (BioCreative II Gene Mention task), disease NER (NCBI Disease corpus), and chemical/drug NER (BC5CDR corpus). These datasets use expert annotators (biologists, pharmacists) and are substantially harder than general-domain NER because entity boundaries in biomedical text are often ambiguous even to human experts.

NER system performance on CoNLL 2003 has improved dramatically over time, driven by architectural advances. Here is how major model families compare:

The jump from CRF to BiLSTM-CRF marks the move to neural representations: instead of hand-engineered features (capitalization, prefix/suffix patterns, gazetteers), the model learns distributed representations of words and their contexts. The jump from BiLSTM-CRF to BERT marks the move to pre-trained language model representations, which encode much richer contextual information about entity mentions.

Not all entity types are equally easy. Let's look at a simulated breakdown to understand the pattern:

Person names typically achieve the highest F1 because they have strong surface-level signals: capitalization, relatively stable name patterns, and extensive coverage in name lists (gazetteers). MISC entities are hardest because the category is semantically heterogeneous. Organizations fall in between: many are easy (well-known company names), but new or ambiguous organizations (a local business, a rarely mentioned agency) are much harder.

Modern NER is dominated by transformer-based approaches. Let's understand exactly how BERT is adapted for NER and implement a complete fine-tuning pipeline.

One subtlety in applying BERT to NER is that BERT uses subword tokenization (WordPiece), which can split words into multiple tokens. The word "Nadella" might become ["Na", "##della"]. When BERT assigns a label to each subword, we need a strategy for aggregating or ignoring subword predictions.

The standard approach is to label only the first subword of each original word and ignore the labels for continuation subwords during evaluation. This keeps the subword details inside the model while maintaining word-level predictions at output time.

The special tokens ([CLS] and [SEP]) get word_id None and are excluded from NER evaluation. For continuation subwords (marked with ## in WordPiece), the standard practice during training is to assign them a special ignore index so they contribute no gradient.

The BERT-for-NER architecture is straightforward. The model takes a tokenized sentence, runs it through BERT's transformer layers to produce a contextualized embedding for each token, and then passes those embeddings through a linear layer that maps to the number of possible BIO tags.

The architecture is simple, but its power comes from the BERT encoder. The 12 transformer layers in BERT-base have already learned, through pre-training on billions of words, to produce representations that encode:

• Co-reference context: whether "he" or "she" in context refers to a named person
• Syntactic structure: whether a token is the head of a noun phrase or a modifier
• World knowledge: associations between particular tokens and entity types

These learned representations mean the linear classification head only needs to learn a relatively simple mapping from representation to label, rather than learning everything from scratch.

In practice, fine-tuning BERT for NER uses the cross-entropy loss on the first subword of each original word, ignoring special tokens and continuation subwords:

The loss ignores the padding positions (where labels are -100) and computes cross-entropy only on the tokens where we have meaningful labels. After a few thousand gradient steps on annotated NER data, the linear classifier learns to combine BERT's contextual representations into accurate entity type predictions.

NER is a global problem. Text in dozens of languages needs entity extraction for practical applications, but annotated training data exists for only a small fraction of the world's languages. Cross-lingual NER addresses this gap by using high-resource language data to build models that work in low-resource languages.

The key enabler for cross-lingual NER is multilingual pre-training. Models like multilingual BERT (mBERT) and XLM-RoBERTa are pre-trained on text from 100+ languages simultaneously using a shared vocabulary and shared transformer parameters. A remarkable finding from the 2019 paper by Pires et al. is that these models develop cross-lingual representations: the representation of "Paris" in French text is close to the representation of "Paris" in English text, even though the model was never explicitly trained to align representations across languages.

This cross-lingual alignment enables zero-shot cross-lingual transfer: train a NER model on English CoNLL 2003 data, and the model generalizes to French, German, Spanish, or Dutch NER without any additional training data in those languages. In the original mBERT experiments, zero-shot transfer achieved 50-70% of fully supervised performance depending on the language, which is remarkable given that no target-language training data was used at all.

XLM-RoBERTa, trained on a larger multilingual corpus (2.5 TB of CommonCrawl data in 100 languages), substantially improves on mBERT's cross-lingual transfer performance. On the XTREME multilingual NER benchmark, XLM-RoBERTa achieves about 65 F1 in zero-shot transfer settings, compared to 58-62 F1 for mBERT.

Even with strong multilingual models, some languages pose particular challenges for NER:

Agglutinative languages like Finnish, Turkish, and Korean attach multiple morphological suffixes to a single word stem. In Finnish, "Helsingissä" (in Helsinki) and "Helsingin" (Helsinki's) are different surface forms of the same city name. Subword tokenization helps here by splitting at morpheme boundaries, but the model still needs to handle many surface variants of the same entity.

Languages without word boundaries like Chinese and Japanese require joint tokenization and NER, since there are no spaces between tokens. Chinese NER typically processes character by character, with the BIO labels assigned to characters rather than words.

Languages with grammatical case like German, Russian, and Czech change the surface form of entity mentions depending on syntactic role. "Angela Merkel" in nominative German versus "Angela Merkels" in genitive German or "Angela Merkel" in accusative German are the same entity in different grammatical forms. A model must recognize all forms as referring to the same person.

Right-to-left scripts like Arabic and Hebrew require special handling for text direction and present additional challenges because Arabic script uses connected letters with no word-internal spacing at all.

For truly low-resource languages with little or no annotated NER data, several techniques help:

Distant supervision automatically labels text by matching strings against existing knowledge bases (like Wikidata). If "Berlin" appears in Wikidata as a city, any occurrence of "Berlin" in text is automatically labeled LOC. Distant supervision is noisy (it cannot handle ambiguous names like "Springfield" or "Amazon"), but it can produce large amounts of weakly labeled training data at low cost.

Cross-lingual data augmentation translates annotated training data from a high-resource language into the target language using machine translation, then applies the original annotations to the translated text. This assumes entity mentions are preserved through translation, which is approximately true for most proper names.

Few-shot learning approaches, including prompt-based methods and retrieval-augmented methods, use small numbers of labeled examples to adapt a pre-trained model to a new domain or language. These methods are particularly relevant for NER because annotating even 50-100 sentences per entity type can significantly improve performance over zero-shot baselines.

The HuggingFace transformers library provides a complete ecosystem for BERT-based NER. Let's walk through loading a pre-trained NER model and applying it to text:

The aggregation_strategy="simple" parameter merges consecutive tokens of the same entity type into a single span, combining their scores by averaging. This is essential for multi-token entities like "Christine Lagarde" that the model labels token by token.

BERT-based NER models produce a softmax probability over entity types for each token. The scores reported by the pipeline are these probabilities for the predicted class. A score of 0.99 indicates the model is very confident; a score of 0.65 indicates uncertainty.

Low-confidence predictions deserve scrutiny. They often correspond to:

• Ambiguous entity names that could be multiple types (is "Apple" the company or the fruit?)
• Unusual or rare entities not well represented in training data
• Entities at the boundary of entity spans (the last token of a long entity often has lower confidence)
• Domain mismatch between training data and the input text

In production systems, you can use confidence thresholds to accept or reject predictions. A high-precision mode might only accept predictions above 0.95, while a high-recall mode might accept anything above 0.7.

While transformer models have largely replaced hand-crafted features, gazetteers and external knowledge sources remain valuable as augmentation, particularly in specialized domains.

A gazetteer is a curated list of entity mentions associated with entity types. For example:

• A gazetteer of company names from a financial database, updated daily
• A gazetteer of drug names from PubChem or DrugBank
• A gazetteer of geographic names from GeoNames

Gazetteers can be used in several ways. The simplest approach is post-processing: after a model makes predictions, override low-confidence predictions if the token matches a gazetteer entry with high reliability. A more principled approach uses gazetteer membership as an additional input feature, either by appending an embedding derived from the gazetteer lookup to the token representation, or by adding a Boolean feature indicating gazetteer match.

Gazetteers are most valuable when the entity vocabulary is well-defined and stable, and when the model encounters terms that are underrepresented in its pre-training data. For general-purpose NER on standard newswire text, gazetteers add little beyond what a well-trained transformer already knows. In specialized domains like pharmaceutical NER, where every approved drug name must be correctly extracted, gazetteer augmentation is often essential.

Let's walk through a complete NER workflow using spaCy. This shows entity extraction, visualization, and custom entity addition.

The spacy.explain() function returns a human-readable description of each label code, which is useful when working with unfamiliar tagsets. Notice that "Thursday" is labeled ORDINAL here, which is a common source of confusion: spaCy's OntoNotes-based model has no explicit DATE label for standalone day names in some contexts.

spaCy's EntityRuler allows you to add rule-based entity detection on top of the statistical model. This is useful for entities that are reliably identifiable by their form (product codes, medical identifiers, stock tickers) but may be missed or mislabeled by the statistical model.

The EntityRuler correctly captures the drug names and the compound ticker pattern, which the base statistical model would likely miss or mislabel. The before="ner" placement means the ruler runs first, and its matches take precedence over the statistical model's predictions for those spans.

Now let's use our metric function from earlier to evaluate predictions against a manually annotated gold standard:

Understanding why a NER system fails is often more valuable than improving the overall F1 score. Systematic error analysis identifies the specific failure modes present in a deployment context, which guides the most effective improvements.

Out-of-vocabulary entities. Any entity name that was absent from the training corpus will be harder to identify. Neural models mitigate this through subword tokenization and contextual representations, but an entity with an entirely novel name and no contextual cues (appearing in a novel context like "Join us for the XTC9000 launch event") will still challenge any model.

Ambiguous surface forms. Many strings that look like entity names are not entities in context. "Amazon" is most often a company, but in an article about rainforest ecology it is a river. "Washington" can be a city, a state, a person, or a university. The model must use context to disambiguate, and when context is uninformative the model must rely on priors learned from training data.

Entity type confusion in rare categories. MISC is the most confused category precisely because it is defined negatively (things that are not PER, ORG, or LOC). Nationality adjectives, events, works of art, and languages all fall into MISC, and the model must learn a heterogeneous set of surface patterns for a single label.

Long entity spans. Entities with many tokens are more prone to boundary errors simply because each additional token is an opportunity for an incorrect prediction. "The International Monetary Fund" has five tokens; each must be correctly tagged for the entity to be counted as correct.

The confusion matrix reveals that MISC is the hardest type both to identify (18% missed) and to classify correctly (confusion with ORG and LOC). This pattern is typical of real NER evaluations. When improving a system, the most productive interventions for MISC accuracy are adding more training examples for MISC types and potentially splitting the MISC category into more specific subtypes (nationalities, events, works of art) if the downstream application can benefit from finer granularity.

When debugging NER errors in production, it is useful to distinguish between three root causes:

The first cause is representation failure: the model has never seen anything similar to this entity in training, so its representation is uninformative. The remedy is to add similar examples to the training data or to use a larger pre-trained model with broader coverage.

The second cause is context failure: the model sees a token that could be an entity but the surrounding context is insufficient to determine the type. For example, "Smith" in isolation could be a person name, a place name, or a common noun. The remedy is to ensure training data includes diverse contextual patterns for each entity type.

The third cause is label noise: the gold annotation itself is wrong or inconsistent. If "Smith" is annotated as PER in 90% of training examples but O in 10% (because annotators disagreed about whether to tag common surnames without first names), the model learns an inconsistent signal. The remedy is to audit training data quality and resolve annotation disagreements systematically.

NER systems are useful, but they come with important limitations that practitioners must understand before deploying them.

Domain shift is the dominant failure mode. A model trained on Reuters newswire will perform well on similar financial and political news, but will degrade substantially on social media posts, clinical notes, legal contracts, or scientific papers. The vocabulary of entity mentions, the syntactic patterns surrounding them, and even the entity type taxonomy may all differ. Before deploying a general-purpose NER model in a specialized domain, you should evaluate it on in-domain annotated examples. The gap between out-of-domain and in-domain performance is often 15-25 F1 points.

Rare and emerging entities are systematically missed. NER models, even transformer-based ones, rely heavily on the memorized associations between surface forms and entity types learned during pretraining. A new company that did not exist during pretraining, a recently coined term, or a person with an unusual name will be missed or mislabeled more often than familiar entities. This is particularly problematic for news applications where the most newsworthy entities are often the least familiar ones.

Annotation inconsistencies propagate to system outputs. Many NER benchmarks contain annotation disagreements, especially for borderline cases (is "the Internet" a MISC entity or not?). Systems trained on inconsistently annotated data learn the pattern of inconsistency, not a principled definition of entity types. When two annotators would disagree on a case, the model's prediction on that case is essentially random.

Evaluation scores can be misleading. Exact match F1 on CoNLL 2003 is a useful benchmark for comparing research systems, but it correlates imperfectly with downstream task performance. A question-answering system that uses NER as a preprocessing step may tolerate boundary errors well but be sensitive to type errors. Reporting system performance using the CoNLL F1 benchmark without also evaluating on the target domain is a common source of overoptimism.

Computational cost of large models. While transformer-based NER achieves the best accuracy, it requires substantially more computation than CRF-based systems. A BERT-base NER model processes roughly 50-100 sentences per second on a modern CPU, while a CRF can process thousands. For high-throughput applications (processing millions of documents daily), the computational cost of transformer-based NER is a real constraint, and practitioners often use lighter models (DistilBERT, smaller spaCy models) with acceptable accuracy tradeoffs.

Despite these limitations, NER has proven enormously useful in practice. It enables structured information extraction at scale: monitoring news for mentions of specific companies, extracting clinical concepts from medical records, building knowledge graphs from text corpora, and improving search engine query understanding. Transformer-based NER models are accurate enough for many production applications, and the field continues to develop better approaches to few-shot learning, cross-domain generalization, and nested entity recognition.

Named Entity Recognition identifies and classifies real-world entity mentions in text. The key concepts from this chapter are:

• Entity types are organized into taxonomies. Core categories (PER, ORG, LOC) appear in every NER system; extended and domain-specific categories are tailored to application needs. The choice of taxonomy reflects assumptions about what downstream consumers of the extracted data need.
• NER is framed as sequence labeling using BIO tagging: each token receives a label encoding both entity type and position within a multi-token span. Any sequence labeling model can be applied to NER through this encoding.
• Boundary detection is hard. Prepositional attachment, titles, possessives, coordination, and abbreviations all create ambiguity about where entity spans begin and end. Annotation guideline inconsistencies compound this difficulty by introducing label noise into training data.
• Nested entities (an entity span containing another entity) cannot be represented in standard BIO tagging. Span-based models or multi-layer annotation schemes are needed for applications that require nested entity extraction.
• NER evolved through three paradigm shifts: rule-based systems using gazetteers and hand-written rules (1980s-1990s), statistical CRF systems using hand-engineered features (1990s-2010s), and neural systems using BiLSTM-CRF and then transformer-based architectures (2015-present). Each shift improved F1 by several points, with the shift to transformers being the most impactful.
• Evaluation uses span-level precision, recall, and F1. Exact match evaluation requires both correct boundaries and correct type; partial match relaxes the boundary requirement. The choice of evaluation scheme can change reported F1 by 5-15 points.
• CoNLL 2003 and OntoNotes 5.0 are the dominant English NER benchmarks. Performance has improved from roughly 82 F1 (CRF) to above 94 F1 (RoBERTa) over two decades.
• Multilingual and cross-lingual NER, enabled by multilingual pre-trained models like XLM-RoBERTa, allows zero-shot transfer to low-resource languages. Agglutinative morphology, case-marked languages, and scriptural variation present additional challenges.
• Domain shift, rare entities, and annotation inconsistencies are the main practical failure modes. Evaluating on in-domain data before deployment is essential. Error analysis should distinguish representation failure, context failure, and label noise as distinct root causes.

The next chapter examines BIO tagging in detail, covering the full family of tagging schemes (BIO, BIOES, BILOU), how to convert between span and tag representations, and how to handle tagging inconsistencies in practice.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about Named Entity Recognition.