Chunking › Training a Statistical Chunker
Language AI HandbookAll chapters Chunking › Training a Statistical ChunkerOne section · Sections in this chapter10 / 15 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Machine learning chunkers learn which IOB tags to assign from annotated examples. Even simple statistical approaches that model the probability of an IOB tag given the current POS tag outperform hand-crafted regex rules.
The simplest statistical model is a unigram tagger: for each POS tag, predict the IOB label that most often followed it in training data.
from nltk.chunk import ChunkParserI
from nltk.tag import BigramTagger, UnigramTagger
class UnigramChunker(ChunkParserI):
"""
Chunker using a unigram model over (POS -> IOB) mappings.
For each POS tag, predicts the most frequent IOB label seen in training.
"""
def __init__(self, train_sents):
# Convert each tree to (pos, iob) pairs for tagger training
train_data = []
for tree in train_sents:
triples = tree_to_iob_triples(tree)
train_data.append([(pos, iob) for _, pos, iob in triples])
self.tagger = UnigramTagger(train_data)
def parse(self, tagged_sent):
"""Assign IOB labels to a POS-tagged sentence."""
pos_tags = [pos for _, pos in tagged_sent]
iob_preds = [self.tagger.tag([pos])[0][1] or "O" for pos in pos_tags]
iob_triples = [
(w, p, iob) for (w, p), iob in zip(tagged_sent, iob_preds)
]
return nltk.chunk.conlltags2tree(iob_triples)
class BigramChunker(ChunkParserI):
"""
Chunker using a bigram model that conditions on the previous POS tag.
Falls back to unigram for unseen bigrams.
"""
def __init__(self, train_sents):
train_data = []
for tree in train_sents:
triples = tree_to_iob_triples(tree)
train_data.append([(pos, iob) for _, pos, iob in triples])
unigram = UnigramTagger(train_data)
self.tagger = BigramTagger(train_data, backoff=unigram)
def parse(self, tagged_sent):
pos_tags = [pos for _, pos in tagged_sent]
tagged_preds = self.tagger.tag(pos_tags)
iob_preds = [iob if iob else "O" for _, iob in tagged_preds]
iob_triples = [
(w, p, iob) for (w, p), iob in zip(tagged_sent, iob_preds)
]
return nltk.chunk.conlltags2tree(iob_triples)
# Train both models
unigram_chunker = UnigramChunker(train_sents)
bigram_chunker = BigramChunker(train_sents)
# Evaluate
unigram_eval = evaluate_chunker(unigram_chunker, test_sents)
bigram_eval = evaluate_chunker(bigram_chunker, test_sents)Statistical Chunker Evaluation Comparison ======================================================= Model Precision Recall F1 -------------------------------------------- Regex 75.75% 72.29% 73.98% Unigram 76.78% 87.77% 81.91% Bigram 81.62% 86.97% 84.21%
The unigram model's improvement over regex is substantial. By learning from thousands of annotated examples, it discovers patterns that hand-crafted rules miss. The bigram model improves further by conditioning on the previous POS tag, capturing sequential patterns like "after a determiner, a noun is likely inside an NP."
Production systems typically use CRF (Conditional Random Field) models, which we cover in the Conditional Random Fields chapter. CRFs can condition on arbitrary features and optimize global sequence consistency, pushing CoNLL-2000 F1 to the low 90s.