Named Entity Recognition › Entity Boundary Detection
Language AI HandbookAll chapters Named Entity Recognition › Entity Boundary DetectionOne section · Sections in this chapter4 / 17 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Identifying where entities begin and end is arguably harder than classifying entity types. Boundary errors account for a large fraction of NER mistakes in practice.
Sources of Boundary Ambiguity
Several linguistic phenomena make boundary detection difficult:
Prepositional attachment. Consider "the University of Michigan professor." Is the entity "University of Michigan" (a named institution) or "University of Michigan professor" (not an entity)? The correct boundary depends on semantic interpretation, not surface syntax.
Possessives and determiners. "Apple's quarterly earnings" contains "Apple" as an ORG entity. Should the possessive "'s" be included? Convention differs across annotation schemes. Most English NER corpora exclude the possessive clitic from the entity span, but this is a choice, not a linguistic necessity.
Titles and honorifics. "President Biden" versus "Biden": is the title part of the entity? The CoNLL 2003 annotation guidelines treat "President Biden" as a single PER span, while some other guidelines treat "Biden" alone as the entity.
Coordination. "Amazon and Google both reported profits" contains two ORG entities, but the surface form shares no obvious boundary marker. Systems trained on isolated entity mentions sometimes fail to split coordinated entities correctly.
Abbreviations. "the U.S." and "the United States" refer to the same entity, but their spans look completely different to a tokenizer.
These ambiguities are not merely academic curiosities. They translate directly into inter-annotator disagreement during corpus construction, and that disagreement propagates into the training data. When two annotators would choose different boundaries for the same span, a model trained on their combined labels learns an inconsistent mapping and produces boundary errors at inference time. Understanding the linguistic sources of boundary ambiguity helps both in designing annotation guidelines and in diagnosing model failures.
Annotation Guidelines and Their Impact
One of the least-discussed but most important factors in NER system quality is the annotation guideline document. Every NER benchmark comes with a guide that specifies, for hundreds of edge cases, exactly which tokens should be included in an entity span and what type label to assign. The CoNLL 2003 annotation guidelines run to dozens of pages. The OntoNotes guidelines are even longer.
These decisions have real consequences. The guideline that says "include titles like President and Dr. in the PER span" means a model trained on CoNLL 2003 will learn to extend person spans to include honorifics, while a model trained on a dataset with a different guideline will learn the opposite behavior. When you use a pre-trained NER model on new data, you are implicitly adopting the annotation conventions of whatever corpus that model was trained on. Mismatches between training conventions and deployment conventions are a common, underappreciated source of NER errors in production.
Nested Entities
One of the most challenging aspects of NER in real-world text is that entities can be nested: one entity span can contain another as a component.
A nested entity occurs when a named entity span is contained within a larger named entity span. For example, in "Bank of America headquarters," the inner span "America" is a LOC entity, while the outer span "Bank of America" is an ORG entity. Both are valid entities, but they overlap.
Standard BIO tagging cannot represent nested entities: a token can have only one tag, so the inner entity must be silently dropped. The sentence "the New York Times" would typically be tagged as a single ORG span, even though "New York" is also a valid LOC entity.
Several approaches address nested NER:
- Layer-by-layer annotation: Annotate one layer of entities at a time, allowing multiple tag sequences per sentence. The ACE 2004 and 2005 datasets use this approach.
- Span-based models: Instead of token-level tagging, enumerate all possible spans and classify each independently. This naturally handles nesting at the cost of candidate spans.
- Constituency parsing integration: Treat nested entities as a variant of the phrase structure parsing problem.
- Graph-based representations: Encode entities as nodes in a graph, with nesting represented by containment edges.
For most practical applications, flat NER (ignoring nesting) is sufficient. Nested NER matters in biomedical information extraction, where phrases like "human immunodeficiency virus type 1 protease inhibitor" contain multiple entity spans of different types.
Let's visualize how nested entities arise:
# Simulate a nested entity scenario using spaCy's span API
import spacy
nlp_blank = spacy.blank("en")
doc_nested = nlp_blank.make_doc(
"The New York Times published an article about Goldman Sachs."
)
# Flat NER would only capture outer spans
flat_entities = [
(0, 4, "ORG"), # "The New York Times"
(8, 10, "ORG"), # "Goldman Sachs"
]
# Nested annotation also captures inner spans
nested_entities = [
(0, 4, "ORG"), # "The New York Times"
(1, 3, "GPE"), # "New York" (inside the ORG)
(8, 10, "ORG"), # "Goldman Sachs"
]
tokens_list = [t.text for t in doc_nested]Tokens: ['The', 'New', 'York', 'Times', 'published', 'an', 'article', 'about', 'Goldman', 'Sachs', '.'] Flat NER spans: [0:4] ORG: 'The New York Times' [8:10] ORG: 'Goldman Sachs' Nested NER spans (including inner entities): [0:4] ORG: 'The New York Times' [1:3] GPE: 'New York' [8:10] ORG: 'Goldman Sachs'
The flat annotation captures "The New York Times" as a single ORG entity, while the nested annotation additionally captures "New York" as a GPE entity contained within it. Standard BIO tagging cannot represent both annotations simultaneously for the same token sequence.