Chunking › IOB Tagging for Chunks
Language AI HandbookAll chapters Chunking › IOB Tagging for ChunksOne section · Sections in this chapter4 / 15 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
As we explored in the BIO Tagging chapter, representing multi-token spans as per-token labels requires an encoding scheme. For chunking, the standard encoding is IOB (Inside-Outside-Beginning) tagging, which uses exactly the same logic as BIO tagging for named entity recognition.
IOB tagging represents chunk boundaries using per-token labels. B-NP marks the first word of a noun phrase, I-NP marks subsequent words in the same NP, and O marks words outside any chunk. The terminology IOB and BIO refer to the same scheme; the order of the letters differs but the encoding is identical.
There is one historical variant worth knowing. The original IOB1 scheme used the B tag only when a chunk immediately followed another chunk of the same type, to distinguish the boundary between two adjacent NPs. The more common IOB2 scheme always uses B for the first token of every chunk, which is simpler and more consistent. All modern benchmarks use IOB2.
Let's see IOB tagging applied to a sentence with multiple chunk types:
# Example sentence with IOB chunk tags
sentence = "The quick brown fox jumps over the lazy dog"
tokens = sentence.split()
# IOB2 tags: B marks the START of every chunk
iob_tags = [
"B-NP", # The (starts NP)
"I-NP", # quick (inside NP)
"I-NP", # brown (inside NP)
"I-NP", # fox (inside NP)
"B-VP", # jumps (starts VP)
"B-PP", # over (starts PP)
"B-NP", # the (starts new NP)
"I-NP", # lazy (inside NP)
"I-NP", # dog (inside NP)
]IOB Chunk Tagging ================================================== Sentence: The quick brown fox jumps over the lazy dog Token IOB Tag Role --------------------------------------------- The B-NP Begin noun phrase quick I-NP Inside noun phrase brown I-NP Inside noun phrase fox I-NP Inside noun phrase jumps B-VP Begin verb phrase over B-PP Begin prep phrase the B-NP Begin noun phrase lazy I-NP Inside noun phrase dog I-NP Inside noun phrase Chunks identified: [The quick brown fox] -> NP [jumps] -> VP [over] -> PP [the lazy dog] -> NP
Converting Between Chunks and IOB Tags
In practice, you often need to convert between two representations: span annotations (start index, end index, type) and per-token IOB sequences. Building on the converter utilities from the BIO Tagging chapter, let's implement these conversions for chunks:
def chunks_to_iob(tokens, chunks):
"""
Convert chunk spans to IOB2 tags.
Args:
tokens: List of token strings
chunks: List of (start_idx, end_idx, chunk_type) tuples
where end_idx is exclusive (Python slicing convention)
Returns:
List of IOB2 tags, one per token
"""
tags = ["O"] * len(tokens)
for start, end, chunk_type in sorted(chunks, key=lambda x: x[0]):
if start < 0 or end > len(tokens) or start >= end:
continue
tags[start] = f"B-{chunk_type}"
for i in range(start + 1, end):
tags[i] = f"I-{chunk_type}"
return tags
def iob_to_chunks(tokens, tags):
"""
Extract chunk spans from IOB2 tags.
Returns:
List of (start_idx, end_idx, chunk_type, text) tuples
"""
chunks = []
current_chunk = None # (start_idx, chunk_type)
for i, (token, tag) in enumerate(zip(tokens, tags)):
if tag.startswith("B-"):
# Close any open chunk first
if current_chunk is not None:
start, ctype = current_chunk
chunks.append((start, i, ctype))
# Start new chunk
current_chunk = (i, tag[2:])
elif tag.startswith("I-"):
chunk_type = tag[2:]
# Handle orphan I tag (I without preceding B)
if current_chunk is None:
current_chunk = (i, chunk_type)
elif current_chunk[1] != chunk_type:
# Type mismatch: close old, start new
start, ctype = current_chunk
chunks.append((start, i, ctype))
current_chunk = (i, chunk_type)
else: # O tag
if current_chunk is not None:
start, ctype = current_chunk
chunks.append((start, i, ctype))
current_chunk = None
# Handle chunk at end of sequence
if current_chunk is not None:
start, ctype = current_chunk
chunks.append((start, len(tokens), ctype))
# Attach text spans
result = []
for start, end, ctype in chunks:
text = " ".join(tokens[start:end])
result.append((start, end, ctype, text))
return result
# Test round-trip conversion
test_tokens = "The quick brown fox jumps over the lazy dog".split()
test_chunks_input = [
(0, 4, "NP"), # The quick brown fox
(4, 5, "VP"), # jumps
(5, 6, "PP"), # over
(6, 9, "NP"), # the lazy dog
]
test_iob = chunks_to_iob(test_tokens, test_chunks_input)
recovered = iob_to_chunks(test_tokens, test_iob)Chunk <-> IOB Round-Trip Conversion ======================================================= Original chunks: [0:4] 'The quick brown fox' -> NP [4:5] 'jumps' -> VP [5:6] 'over' -> PP [6:9] 'the lazy dog' -> NP Generated IOB tags: The -> B-NP quick -> I-NP brown -> I-NP fox -> I-NP jumps -> B-VP over -> B-PP the -> B-NP lazy -> I-NP dog -> I-NP Recovered chunks: [0:4] 'The quick brown fox' -> NP [4:5] 'jumps' -> VP [5:6] 'over' -> PP [6:9] 'the lazy dog' -> NP Round-trip match: True
The round-trip works because IOB2 uniquely encodes every possible chunk configuration. The only ambiguity in the decoding step is handling "orphan I" tags (an I tag without a preceding B of the same type), which we resolve by treating the I as implicitly starting a new chunk.