Chunking › What Is Chunking?

Chunking identifies contiguous spans of tokens that form syntactic units. Unlike full parsing, which produces tree structures with unlimited nesting, chunking produces a flat sequence of labeled segments. Each token belongs to exactly one chunk, or no chunk at all if it falls outside every phrase boundary.

The word "shallow" in "shallow parsing" refers to the depth of structural analysis. Chunking reads the surface of a sentence and identifies phrase-like groups, but it does not resolve questions like whether a prepositional phrase modifies the subject or the verb. Those decisions require deeper semantic understanding, and chunking deliberately avoids them.

Consider the sentence "The black cat sat on the mat." A chunker might produce:

• [The black cat]$_{NP}$, noun phrase (subject)
• [sat]$_{VP}$, verb phrase (predicate)
• [on]$_{PP}$, prepositional phrase marker
• [the mat]$_{NP}$, noun phrase (object of preposition)

The key properties of chunking are:

• Non-overlapping: Each word belongs to at most one chunk. Two chunks cannot share a token.
• Non-recursive: Chunks do not contain other chunks of the same type. You won't find an NP inside an NP in a flat chunking annotation.
• Contiguous: Chunks consist of consecutive words. A chunk cannot span across a gap.
• Partial coverage: Some words may not belong to any chunk. Articles that introduce a phrase belong to the phrase, but punctuation often remains unchunked.

Let's see chunking in action with NLTK. The ne_chunk function identifies named entities as chunks, which is a specific form of chunking:

NLTK's ne_chunk function returns a tree where named entities appear as labeled subtrees. Each subtree is a chunk. This pattern, where a sequence labeling task produces labeled spans, is the core idea behind all chunking.