Part-of-Speech Tagging › POS Tagging for Downstream Tasks
Language AI HandbookAll chapters Part-of-Speech Tagging › POS Tagging for Downstream TasksOne section · Sections in this chapter10 / 18 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
POS tags are rarely the end goal. They serve as features or intermediate representations for richer NLP tasks.
Improving Named Entity Recognition
Named entity recognizers benefit from POS tags because entity names follow characteristic POS patterns. A person's name is typically NNP NNP (proper noun proper noun). An organization name often ends with NNP NNP NNP. These patterns help the NER system focus its attention:
def find_proper_noun_sequences(doc):
"""Find sequences of proper nouns as candidate named entities."""
candidates = []
current_seq = []
for token in doc:
if token.pos_ == "PROPN":
current_seq.append(token.text)
else:
if len(current_seq) >= 1:
candidates.append(" ".join(current_seq))
current_seq = []
if current_seq:
candidates.append(" ".join(current_seq))
return candidatesProper Noun Sequence Detection (Candidate NER) ================================================================= Text: Barack Obama visited the United Nations headquarters in New York City. Candidates: ['Barack Obama', 'United Nations', 'New York City'] Text: Apple Inc. reported record revenue, outpacing Microsoft and Amazon. Candidates: ['Apple Inc.', 'Microsoft', 'Amazon'] Text: The World Health Organization published new guidelines last Tuesday. Candidates: ['World Health Organization', 'Tuesday']
This heuristic is not perfect (it will miss multi-word entities that include non-proper-nouns), but it shows how POS tags provide a cheap first approximation for entity detection.
Syntactic Pattern Matching
POS tags enable pattern-based information extraction. Noun chunks (sequences of determiners, adjectives, and nouns) identify the subjects and objects of sentences. Verb phrases identify actions. This structured extraction is faster and more interpretable than neural extraction for well-defined patterns:
def extract_verb_arg_pairs(doc):
"""Extract (subject, verb, object) triples using dependency + POS."""
triples = []
for token in doc:
if token.pos_ == "VERB":
subj = [
c.text
for c in token.children
if c.dep_ in ("nsubj", "nsubjpass")
]
obj = [
c.text
for c in token.children
if c.dep_ in ("dobj", "obj", "pobj")
]
if subj and obj:
triples.append((subj[0], token.lemma_, obj[0]))
return triplesVerb-Argument Extraction ================================================== Text: Scientists discovered a new exoplanet orbiting a distant star. (Scientists, discover, exoplanet) Text: The committee approved the budget after reviewing the proposals. (committee, approve, budget) Text: Researchers published their findings in Nature. (Researchers, publish, findings)
Morphological Analysis and Lemmatization
POS tags are essential for correct lemmatization. The word "saw" has two lemmas depending on its POS: as a verb (VBD), it lemmatizes to "see"; as a noun (NN), it lemmatizes to "saw" (a cutting tool). Without POS information, a lemmatizer must guess:
POS-Dependent Lemmatization
-------------------------------------------------------
'saw' in "I saw the movie."
POS: VERB | Lemma: see
'saw' in "She picked up the saw."
POS: NOUN | Lemma: saw
'fall' in "The leaves turn color in fall."
POS: NOUN | Lemma: fall
'fall' in "The leaves fall in October."
POS: VERB | Lemma: fallThis dependency on POS for correct lemmatization propagates through the entire NLP pipeline. A wrong POS tag early in the pipeline causes wrong lemmatization, which causes wrong feature matching downstream.
Shallow Parsing and Chunking
One of the most direct uses of POS tags is shallow parsing, also known as chunking. A chunker groups consecutive tokens into non-overlapping phrases, identifying noun phrases, verb phrases, and prepositional phrases without building a full recursive parse tree.
The most common chunking task is noun phrase chunking (also called base noun phrase chunking or NP chunking). A base NP is a simple, non-recursive noun phrase: the outermost noun phrase that does not itself contain any other noun phrases. For example, in "the quick brown fox jumps over the lazy dog," the base NPs are "the quick brown fox" and "the lazy dog." The verb phrase "jumps over" and the prepositional phrase "over the lazy dog" are not base NPs.
POS tags provide the necessary information for NP chunking. The pattern for an English base NP is roughly: an optional determiner, zero or more adjectives, and one or more nouns. In Penn Treebank tags, this pattern is DT? JJ* NN+ (where ? means zero or one, * means zero or more, and + means one or more). A simple regex over the tag sequence captures this structure:
import re
def chunk_np(tagged_sentence):
"""Extract noun phrases from a tagged sentence using a regex over POS tags."""
# Join tags into a string for regex matching
tag_string = " ".join(tag for _, tag in tagged_sentence)
words = [word for word, _ in tagged_sentence]
# Simple NP pattern: optional DT, zero or more JJ/JJR/JJS/CD, then NN/NNS/NNP/NNPS
np_pattern = re.compile(
r"(?:DT\s)?(?:(?:JJ[RS]?|CD)\s)*(?:(?:NN[SP]?S?)\s?)+"
)
noun_phrases = []
for m in np_pattern.finditer(tag_string + " "):
# Convert character offset to token offset
start_char = m.start()
tag_prefix = tag_string[:start_char]
token_start = len(tag_prefix.split()) if tag_prefix.strip() else 0
num_tokens = len(m.group().split())
np_words = words[token_start : token_start + num_tokens]
noun_phrases.append(" ".join(np_words))
return noun_phrasesNoun Phrase Chunking via POS Patterns
============================================================
Sentence: The quick brown fox jumps over the lazy dog
Tagged: [('The', 'DT'), ('quick', 'JJ'), ('brown', 'JJ'), ('fox', 'NN'), ('jumps', 'VBZ'), ('over', 'IN'), ('the', 'DT'), ('lazy', 'JJ'), ('dog', 'NN')]
spaCy NP chunks: ['The quick brown fox', 'the lazy dog']
Sentence: Three large pharmaceutical companies announced new clinical trials
Tagged: [('Three', 'CD'), ('large', 'JJ'), ('pharmaceutical', 'JJ'), ('companies', 'NNS'), ('announced', 'VBD'), ('new', 'JJ'), ('clinical', 'JJ'), ('trials', 'NNS')]
spaCy NP chunks: ['Three large pharmaceutical companies', 'new clinical trials']
Sentence: The Federal Reserve raised interest rates by half a percentage point
Tagged: [('The', 'DT'), ('Federal', 'NNP'), ('Reserve', 'NNP'), ('raised', 'VBD'), ('interest', 'NN'), ('rates', 'NNS'), ('by', 'IN'), ('half', 'PDT'), ('a', 'DT'), ('percentage', 'NN'), ('point', 'NN')]
spaCy NP chunks: ['The Federal Reserve', 'interest rates', 'half a percentage point']Chunking with POS patterns is fast and requires no additional training data beyond the POS tagger itself. The resulting noun phrases can be used directly as features for downstream tasks. In information extraction, noun phrase chunks are candidate argument slots for relation extraction. In text summarization, high-frequency noun phrases identify the key entities being discussed. In search engine indexing, noun phrase chunking extracts multi-word key terms that are more informative than individual words.
The limitation of pattern-based chunking is that it handles only the simplest phrase structures. Complex noun phrases containing relative clauses ("the company that won the contract"), appositive phrases ("Apple Inc., the Cupertino-based technology giant"), or coordination ("cats and dogs") require recursive parsing or more sophisticated models.