Chunking › Summary

Chunking identifies non-overlapping, non-recursive phrase segments in text. This provides a useful middle ground between POS tagging and full syntactic parsing. The key concepts from this chapter are:

• Chunk types include noun phrases (NP), verb phrases (VP), and prepositional phrases (PP). NP chunking is the most widely studied and practically applied, capturing the noun-headed phrases that carry most of the content in a sentence.
• IOB tagging encodes chunk boundaries as per-token labels: B marks the first token of a chunk, I marks continuation tokens, and O marks tokens outside all chunks. IOB2, where B always marks the first token of every chunk, is the modern standard and is simpler to work with than the original IOB1 variant.
• Chunking vs. parsing represents a deliberate tradeoff: chunking is faster and avoids hard attachment ambiguities that require semantic knowledge to resolve. Full parsing captures complete hierarchical structure but is slower and more error-prone.
• Regex-based chunking with NLTK's RegexpParser defines patterns over POS tag sequences. It is intuitive and transparent but limited to local patterns and sensitive to POS errors. Chinking (the }{ syntax) provides an alternative by specifying what should be excluded from chunks.
• Statistical chunkers learn IOB label distributions from annotated data. Even simple unigram and bigram models trained on CoNLL-2000 substantially outperform hand-crafted rules. Production systems use CRF models that condition on arbitrary features and optimize global sequence consistency.
• Practical applications include subject-verb-object extraction, noun phrase keyword mining, and answer candidate identification for question answering. Chunking provides lightweight structural annotations that simplify downstream processing without the complexity of full parsing.

The key parameters and syntax for chunking tools are:

NLTK RegexpParser grammar syntax:

• {<pattern>}: Include matching tokens in a chunk (chunking)
• }<pattern>{: Remove matching tokens from a chunk (chinking)
• <DT|PRP$>: Match either DT or PRP$ POS tags (alternation)
• <JJ>*: Zero or more JJ tokens (Kleene star)
• <NN.*>+: One or more tokens whose POS starts with NN (plus quantifier with wildcard)

CoNLL-2000 corpus loading:

• chunk_types=["NP", "VP", "PP"]: Specify which chunk types to retain
• chunked_sents("train.txt"): Load as NLTK parse trees
• tree_to_iob_triples(tree): Convert trees to (word, POS, IOB) triples for ML training

spaCy noun chunks:

• doc.noun_chunks: Iterate over all noun phrases in a document
• chunk.root: The head noun of the chunk
• chunk.root.dep_: The syntactic dependency role of the head noun

The next chapters explore the probabilistic sequence models that power production-quality chunkers and other sequence labeling systems, starting with Hidden Markov Models.