Part-of-Speech Tagging › POS Tagging as a Classification Problem
Language AI HandbookAll chapters Part-of-Speech Tagging › POS Tagging as a Classification ProblemOne section · Sections in this chapter5 / 18 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
POS tagging is a supervised classification problem. Given a word and its context, predict the tag. This framing opens up the full toolkit of machine learning methods.
The challenge is that POS tagging operates at the sequence level: we are not classifying tokens independently but finding the most probable tag sequence for an entire sentence. The tags interact, because knowing that one word is a determiner strongly constrains the possible tags for the following word (determiners are almost always followed by adjectives or nouns).
POS tagging assigns a tag to every token in a sequence simultaneously, not independently. The optimal tag for each token depends on the tags of its neighbors. This is why sequence models (like HMMs and CRFs) outperform independent classifiers.
The two main approaches to capturing these sequence dependencies are:
- Generative models: Model the joint probability and find the tags that maximize it. Hidden Markov Models (HMMs) are the classic example.
- Discriminative models: Model directly, conditioning on all observable information. Conditional Random Fields (CRFs) and neural sequence models take this approach.
We will cover HMMs and CRFs in depth in subsequent chapters. Here we focus on the features and evaluation principles that apply across methods.
Features for POS Tagging
What information helps predict a word's POS tag? Feature engineering reveals what contextual signals are informative:
Word identity: The word itself is the strongest single feature. "The" is almost always a determiner. "Quickly" is almost always an adverb.
Suffix features: Morphological endings are highly predictive. Words ending in "-tion" are almost always nouns (perception, relation). Words ending in "-ly" are almost always adverbs (quickly, slowly). Words ending in "-ing" are gerunds or present participles (running, eating).
Prefix features: Less predictive than suffixes but still useful. Words beginning with "un-" are often adjectives or verbs (unhappy, undo).
Capitalization: Capitalized words mid-sentence are likely proper nouns (NNP). Sentence-initial capitalization is uninformative but all-caps words are often abbreviations.
Context window: The tags and words of neighboring tokens. After a determiner, a noun or adjective is expected. After a modal verb like "can," a base-form verb typically follows.
Word shape: Patterns like "Xxxxx" (capitalized word), "XXXXX" (all-caps), "99.99" (decimal number) capture orthographic structure independent of the specific characters.
The relative importance of these features is not uniform across all words. For common, frequent words that appear many times in training data, word identity is by far the most useful feature. For rare or out-of-vocabulary words, morphological features (especially suffixes) become critical. A model that relies entirely on word identity will fail on unseen vocabulary; a model that ignores word identity will fail on common ambiguous words. Good feature engineering combines both.
def extract_features(sentence, i):
"""Extract features for the i-th word in a sentence.
sentence: list of words (strings)
i: index of the target word
Returns a dictionary of feature name -> feature value.
"""
word = sentence[i]
features = {
# Word identity
"word": word.lower(),
"word_upper": word.isupper(),
"word_title": word.istitle(),
# Suffix features (morphological)
"suffix_2": word[-2:].lower() if len(word) >= 2 else word.lower(),
"suffix_3": word[-3:].lower() if len(word) >= 3 else word.lower(),
"suffix_4": word[-4:].lower() if len(word) >= 4 else word.lower(),
# Prefix features
"prefix_2": word[:2].lower() if len(word) >= 2 else word.lower(),
"prefix_3": word[:3].lower() if len(word) >= 3 else word.lower(),
# Lexical properties
"is_digit": word.isdigit(),
"contains_hyphen": "-" in word,
"contains_digit": any(c.isdigit() for c in word),
# Position in sentence
"is_first": i == 0,
"is_last": i == len(sentence) - 1,
}
# Previous word context
if i > 0:
prev_word = sentence[i - 1]
features["prev_word"] = prev_word.lower()
features["prev_word_title"] = prev_word.istitle()
else:
features["prev_word"] = "<START>"
features["prev_word_title"] = False
# Next word context
if i < len(sentence) - 1:
next_word = sentence[i + 1]
features["next_word"] = next_word.lower()
features["next_word_title"] = next_word.istitle()
else:
features["next_word"] = "<END>"
features["next_word_title"] = False
return featuresFeature extraction for 'jumps' (index 4): -------------------------------------------------- contains_digit False contains_hyphen False is_digit False is_first False is_last False next_word over next_word_title False prefix_2 ju prefix_3 jum prev_word fox prev_word_title False suffix_2 ps suffix_3 mps suffix_4 umps word jumps word_title False word_upper False
The suffix features "umps" and "mps" might not look informative for "jumps" in isolation, but when the model sees the suffix "-s" on a word following an adjective-modified noun, it learns that this pattern almost always signals a third-person singular present verb (VBZ). Feature engineering is about giving the model the right vocabulary to express these patterns.
The Feature Interaction Problem
One limitation of feature engineering is that individual features do not capture interactions. The feature "suffix is -ed" fires for both past tense verbs (walked) and past participle adjectives (excited, concerned). The feature alone is ambiguous. But the combination "suffix is -ed AND previous word is 'have' or 'has'" almost uniquely identifies a past participle in perfect aspect. Traditional feature engineering handles this by creating conjunction features, but the number of possible conjunctions grows combinatorially with the number of base features.
This is one key motivation for neural approaches to POS tagging. Rather than manually specifying which feature combinations matter, a neural network learns to form its own internal representations that capture the relevant interactions. A bidirectional LSTM reading the sequence "has walked" will produce a hidden state for "walked" that reflects the presence of the auxiliary "has" in the left context, without any explicit feature engineering. The word embedding for "has" influences the LSTM state, which in turn influences the output distribution over tags for "walked." The model learns the interaction implicitly.