BIO Tagging › BIO Tagging in Transformer-Based Models
Language AI HandbookAll chapters BIO Tagging › BIO Tagging in Transformer-Based ModelsOne section · Sections in this chapter10 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Modern NER systems built on BERT and similar transformers require a careful treatment of BIO tags due to subword tokenization. BERT's WordPiece tokenizer splits words into subword units, so "Washington" might become ["Washington"] or ["Wash", "##ington"] depending on whether it appears in the model's vocabulary. The "##" prefix marks continuation pieces of the same word.
This creates an alignment problem: your BIO tags are defined over words, but the model processes subword tokens. The standard approach is to assign the word's BIO tag to the first subword piece and a special label, typically "-100" in Hugging Face terminology, to all subsequent pieces. The loss function ignores positions labeled "-100," so the model only predicts tags for the first piece of each word.
def align_labels_with_tokens(word_ids, word_labels):
"""
Align word-level BIO labels with subword tokenization.
word_ids: list from tokenizer.word_ids(), maps token position to word index
(None for special tokens like [CLS] and [SEP])
word_labels: list of BIO tags, one per word
Returns label_ids with -100 for special tokens and continuation subwords.
"""
label_ids = []
previous_word_idx = None
for word_idx in word_ids:
if word_idx is None:
# Special token ([CLS], [SEP], [PAD])
label_ids.append(-100)
elif word_idx != previous_word_idx:
# First subword of a new word: use the word's label
label_ids.append(word_labels[word_idx])
else:
# Continuation subword: ignore during loss computation
label_ids.append(-100)
previous_word_idx = word_idx
return label_ids
# Simulate tokenization with subword splitting
# Word: Barack Obama visited New York City
words = ["Barack", "Obama", "visited", "New", "York", "City"]
bio_word_labels = [0, 1, 2, 3, 4, 4] # 0=B-PER, 1=I-PER, 2=O, 3=B-LOC, 4=I-LOC
# Simulated word_ids (e.g., "visited" splits into "visit" + "##ed")
# word_ids: [None, 0, 1, 2, 2, 3, 4, 5, None]
# CLS Br Ob vis ##ed New York City SEP
simulated_word_ids = [None, 0, 1, 2, 2, 3, 4, 5, None]
aligned = align_labels_with_tokens(simulated_word_ids, bio_word_labels)Subword Alignment for Transformer NER: --------------------------------------------- Subword Word ID Label --------------------------------------------- [CLS] None [IGN] Barack 0 B-PER Obama 1 I-PER visit 2 O ##ed 2 [IGN] New 3 B-LOC York 4 I-LOC City 5 I-LOC [SEP] None [IGN] Note: [IGN] positions are excluded from loss computation. The '##ed' piece of 'visited' is ignored since the first subword 'visit' already carries the O label.
This alignment procedure ensures that the model makes one prediction per word (from the first subword) and that subword boundaries do not corrupt the BIO sequence. Without proper alignment, a model might produce a BIO sequence over subword units that is longer than the word sequence, making it impossible to recover coherent entity spans.
The alignment also affects how you interpret predictions at inference time. After the model predicts labels for all subword positions, you collect only the labels at first-subword positions (where word_ids changes to a new value) and discard continuation labels. These first-subword labels form the word-level BIO sequence, which you can then decode using the standard bio_to_spans function we implemented earlier. The subword processing is a preprocessing and postprocessing layer wrapped around the same core BIO machinery.