Named Entity Recognition › NER as Sequence Labeling
Language AI HandbookAll chapters Named Entity Recognition › NER as Sequence LabelingOne section · Sections in this chapter3 / 17 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The cleanest formulation of NER turns it into a token-level classification problem. Rather than predicting entity spans directly, we assign a label to each token and use a tagging scheme that encodes both the entity type and the position of the token within a multi-token span.
This framing has a powerful practical consequence: any sequence labeling model, from a simple logistic regression to a transformer, can be applied to NER without fundamental changes to the model architecture. The NER-specific engineering lives in the tagging scheme and the loss function, not in the model itself.
The BIO Tagging Scheme
The most widely used scheme is BIO (also called IOB2). Each token receives one of three tag prefixes:
- B-TYPE: The token begins an entity of type TYPE.
- I-TYPE: The token is inside a continuing entity of type TYPE.
- O: The token is outside any entity.
For the sentence "Tim Cook visited Paris", the BIO labels would be:
| Token | BIO Tag |
|---|---|
| Tim | B-PER |
| Cook | I-PER |
| visited | O |
| Paris | B-LOC |
This encoding turns span detection into token classification. A model that correctly predicts B-PER then I-PER for adjacent tokens implicitly identifies the span "Tim Cook" as a person entity. Span recovery from BIO tags is straightforward: a new entity begins whenever a B- tag appears or an I- tag follows a different entity type.
The BIO tagging scheme and its variants (BIOES, BILOU) are covered in detail in the next chapter on BIO Tagging. For now, the key insight is that this encoding makes NER amenable to any sequence labeling model: HMMs, CRFs, or neural architectures with token-level classification heads.
From Tokens to Spans
Given BIO predictions, recovering entity spans requires a simple decoding pass:
def bio_to_spans(tokens, bio_tags):
"""Convert a BIO-tagged sequence to a list of (start, end, label) spans."""
spans = []
current_start = None
current_label = None
for i, (token, tag) in enumerate(zip(tokens, bio_tags)):
if tag.startswith("B-"):
# Close any open span
if current_start is not None:
spans.append((current_start, i, current_label))
# Open new span
current_start = i
current_label = tag[2:]
elif tag.startswith("I-"):
label = tag[2:]
# If I- follows a different type or O, treat as a new B-
if current_label != label:
if current_start is not None:
spans.append((current_start, i, current_label))
current_start = i
current_label = label
else: # O
if current_start is not None:
spans.append((current_start, i, current_label))
current_start = None
current_label = None
# Close any trailing span
if current_start is not None:
spans.append((current_start, len(tokens), current_label))
return spans
tokens = ["Tim", "Cook", "visited", "Paris", "and", "London"]
bio_tags = ["B-PER", "I-PER", "O", "B-LOC", "O", "B-LOC"]
spans = bio_to_spans(tokens, bio_tags)Token → Tag mapping: Tim B-PER Cook I-PER visited O Paris B-LOC and O London B-LOC Recovered spans: [0:2] PER: 'Tim Cook' [3:4] LOC: 'Paris' [5:6] LOC: 'London'
The decoder correctly recovers "Tim Cook" as a single PER span (tokens 0-2), "Paris" as LOC (token 3-4), and "London" as LOC (token 5-6). The function also handles the case where an I- tag follows a different entity type (a common tagging inconsistency), treating it as a new begin rather than crashing.