Chunking › IOB1 vs. IOB2: The Two Variants
Language AI HandbookAll chapters Chunking › IOB1 vs. IOB2: The Two VariantsOne section · Sections in this chapter6 / 15 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Before diving into implementation, it's worth understanding the historical variant IOB1, which you'll encounter in older papers and datasets.
In IOB1 (the original scheme), the B prefix marks the first token of a chunk only when the chunk immediately follows another chunk of the same type. Otherwise, the first token of a chunk uses the I prefix. This might seem like an odd design choice, but it minimizes the number of distinct labels needed when adjacent same-type chunks are rare.
Consider two adjacent NPs: "John" followed immediately by "Mary" with no separator:
# Compare IOB1 and IOB2 for adjacent chunks of the same type
example_tokens = ["John", "Mary", "left"]
gold_chunks = [(0, 1, "NP"), (1, 2, "NP"), (2, 3, "VP")]
# IOB2: B always marks the first token of a chunk
iob2_tags = chunks_to_iob(example_tokens, gold_chunks)
# IOB1: B only used when same-type chunk follows same-type chunk
# Adjacent NPs: first NP uses I (since nothing precedes it of same type yet)
# Second NP uses B (because it follows an NP)
iob1_tags = ["I-NP", "B-NP", "I-VP"] # IOB1 encodingIOB1 vs. IOB2 Comparison ================================================== Sentence: John Mary left Chunks: [John]NP [Mary]NP [left]VP (two adjacent NPs) Token IOB2 IOB1 Difference -------------------------------------------------- John B-NP I-NP DIFFER Mary B-NP B-NP left B-VP I-VP DIFFER Key insight: IOB2: B always marks chunk start (simpler, more consistent) IOB1: B only used to disambiguate adjacent same-type chunks Modern practice: use IOB2
IOB2 is strictly easier to work with. Every B tag means "new chunk starts here," regardless of context. For a sequence labeling model learning from data, IOB2 provides cleaner, more consistent targets. If you encounter IOB1-tagged data (some older CoNLL datasets use it), convert to IOB2 before training.