Chunking › Using the CoNLL-2000 Dataset
Michael BrenndoerferUpdated March 21, 20262 min read
Language AI HandbookAll chapters Part 6 of 67Sequence Labeling
Chunking › Using the CoNLL-2000 DatasetOne section · Sections in this chapter8 / 15 ↓
Sections in this chapter15
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The CoNLL-2000 shared task established the standard benchmark for chunking. It uses a portion of the Penn Treebank Wall Street Journal text, annotated with POS tags and IOB chunk labels for three chunk types: NP, VP, and PP.
In[32]:
Codefrom nltk.corpus import conll2000
# Load training and test data
train_sents = conll2000.chunked_sents(
"train.txt", chunk_types=["NP", "VP", "PP"]
)
test_sents = conll2000.chunked_sents("test.txt", chunk_types=["NP", "VP", "PP"])
# Examine the first sentence
sample_sent = train_sents[0]Out[33]:
ConsoleCoNLL-2000 Dataset Overview ==================================================
Training sentences: 8,936
Test sentences: 2,012 Sample sentence (NLTK Tree format): (S (NP Confidence/NN) (PP in/IN) (NP the/DT pound/NN) (VP is/VBZ widely/RB expected/VBN to/TO take/VB) (NP another/DT sharp/JJ dive/NN) if/IN (NP trade/NN figures/NNS) (PP for/IN) (NP September/NNP) ,/, due/JJ (PP for/IN) (NP release/NN) (NP tomorrow/NN) ,/, (VP fail/VB to/TO show/VB) (NP a/DT substantial/JJ improvement/NN) (PP from/IN) (NP July/NNP and/CC August/NNP) (NP 's/POS near-record/JJ deficits/NNS) ./.)
Chunk type frequency (first 1,000 training sentences): NP: 6,211 VP: 2,399 PP: 2,397
Converting CoNLL Data to IOB Format
For machine learning approaches, we need the data in (word, POS, IOB) triple format. This function converts NLTK parse trees into that representation:
In[34]:
Codedef tree_to_iob_triples(tree):
"""Convert an NLTK chunk tree to (word, pos, iob) triples."""
triples = []
for subtree in tree:
if hasattr(subtree, "label"):
chunk_type = subtree.label()
for i, (word, pos) in enumerate(subtree.leaves()):
iob = f"B-{chunk_type}" if i == 0 else f"I-{chunk_type}"
triples.append((word, pos, iob))
else:
# Single token outside any chunk
word, pos = subtree
triples.append((word, pos, "O"))
return triples
# Convert the sample sentence
sample_iob = tree_to_iob_triples(sample_sent)Out[35]:
ConsoleIOB Triple Format (word, POS, IOB) ================================================== Word POS IOB -------------------------------------- Confidence NN B-NP in IN B-PP the DT B-NP pound NN I-NP is VBZ B-VP widely RB I-VP expected VBN I-VP to TO I-VP take VB I-VP another DT B-NP sharp JJ I-NP dive NN I-NP if IN O trade NN B-NP figures NNS I-NP ... (22 more tokens)