Part-of-Speech Tagging: Tag Sets, Algorithms, Implementation

Every word in a sentence carries two kinds of information: its meaning and its grammatical role. Part-of-speech tagging (POS tagging) is the task of automatically assigning each word in a sentence a label that captures that grammatical role. Given "The quick brown fox jumps over the lazy dog," a POS tagger produces output like: DET ADJ ADJ NOUN VERB ADP DET ADJ NOUN.

Why does this seemingly simple annotation matter so much? POS tags give downstream NLP systems a compact, structured summary of sentence structure without the full overhead of a complete syntactic parse. Syntactic parsers use POS tags to prune the search space of possible trees. Named entity recognizers use them to identify entity boundaries, since proper noun sequences (NNP NNP) are strong signals for person or organization names. Question answering systems use verb tags to locate the action a question is asking about. Even modern neural language models, which learn representations end-to-end without explicit POS supervision, benefit from POS auxiliary tasks during training.

The importance of POS tagging extends beyond these direct applications. POS labels function as a compact encoding of local syntactic context. A model that knows a given token is a past-tense verb immediately knows something important: the sentence likely has a predicate at this position, the argument structure involves a subject and possibly an object, and the temporal framing of the clause is in the past. Encoding all of that information through a single two- or three-character tag is extraordinarily economical. For classical NLP pipelines built around hand-crafted feature engineering, POS tags were not just useful; they were indispensable.

This chapter covers the full arc of POS tagging: the tag sets NLP practitioners use, the deep ambiguity problem that makes tagging non-trivial, how both rule-based and statistical approaches tackle that ambiguity, and how we evaluate and implement taggers in practice. By the end you will have working code for both training and applying POS taggers and a clear mental model of where accuracy gains come from.

Parts of speech, also called word classes or lexical categories, group words by their grammatical function and syntactic behavior. The classification dates to ancient Greek grammarians and was systematized by Pāṇini for Sanskrit around 400 BCE. Modern linguistics recognizes that word classes are defined by distributional properties: words that occupy the same syntactic positions and combine with the same kinds of modifiers belong to the same class.

The traditional list of eight parts of speech taught in school (noun, verb, adjective, adverb, pronoun, preposition, conjunction, interjection) is a coarse-grained starting point. Computational linguistics requires finer distinctions. Instead of a single "verb" category, we need to distinguish base form verbs, past tense verbs, gerunds, past participles, and third-person singular present forms, because they behave differently in syntax and morphology.

The core intuition behind any POS system is that words group into categories based on:

• Distributional behavior: Nouns follow determiners ("the dog"), verbs follow noun phrases ("the dog barked")
• Morphological properties: Nouns form plurals (-s), verbs inflect for tense (-ed, -ing), adjectives form comparatives (-er)
• Semantic tendencies: Nouns typically name entities, verbs typically name events, adjectives name properties

These three criteria often agree but sometimes conflict, which is why POS classification requires careful linguistic thought.

It is worth pausing on why distributional behavior is the most reliable of these three criteria. Consider the word "table." Semantically it names a physical object, so you might guess it is always a noun. But "table the motion" is perfectly grammatical English where "table" functions as a verb. The semantic criterion would lead you astray. The distributional criterion works better: in "table the motion," "table" appears immediately after a subject noun phrase and before a direct object noun phrase, which is a verb position. The morphological criterion is also informative but incomplete: "table" has no distinctive verb morphology here, but English is a low-inflection language where verbs frequently appear in their bare stem form.

Computational systems that learn POS categories from annotated corpora are, implicitly, learning these distributional patterns. When a neural tagger encounters a word it has never seen before, its prediction is guided by the context words that surround it, not by any innate category knowledge. This distributional learning is one of the key mechanisms that allows modern taggers to handle out-of-vocabulary words gracefully.

A tag set defines the inventory of labels a tagger assigns. The choice of tag set involves a fundamental tradeoff: more tags allow finer-grained distinctions that benefit some downstream tasks, but require more annotated data to learn reliably and are harder to assign consistently.

The Penn Treebank (PTB) tag set, developed at the University of Pennsylvania as part of the Penn Treebank project in the early 1990s, became the standard for English NLP. It contains 36 POS tags plus 12 punctuation and symbol tags.

The fine-grained verb distinctions matter considerably in practice. Knowing that "barked" is VBD (past tense) versus "barking" is VBG (gerund/participle) tells a parser whether to attach it as the main verb or as a modifier. The six verb tags capture the full English verbal paradigm and allow downstream systems to reason about tense and aspect.

The Penn Treebank tag set was built around the Wall Street Journal corpus and reflects the characteristics of formal written English. This origin has consequences: the tag set handles financial text well but struggles with informal registers, social media text, and domain-specific terminology.

One design decision in the PTB tag set deserves particular attention: the treatment of "IN" as a single tag covering both prepositions and subordinating conjunctions. This conflation is linguistically controversial. "After dinner" uses "after" as a preposition governing a noun phrase, while "after she left" uses "after" as a subordinating conjunction introducing a clause. These are syntactically distinct, but they were collapsed into a single tag because reliable disambiguation requires the kind of deep syntactic analysis the tag set was meant to enable in the first place. This circular dependency is a recurring theme in annotation design: the tags you want to assign depend on information that the tags are supposed to provide.

The PTB tag set is designed for English, and extending it to other languages is problematic. German uses case inflections that English lacks. Mandarin Chinese has no morphological inflection at all. Arabic has a complex system of verb derivations. The Universal Dependencies (UD) project introduced a cross-linguistic tag set with 17 UPOS tags designed to apply consistently across languages.

The 17-tag set achieves cross-linguistic applicability by collapsing distinctions that are English-specific. All six English verb inflection tags (VB, VBD, VBG, VBN, VBP, VBZ) merge into VERB. The key insight is that morphological distinctions like tense can be captured through additional morphological annotation layers (called morphological features in UD) rather than proliferating the POS tag set itself.

The UD project has annotated over 200 treebanks across more than 100 languages using this framework, making it the largest cross-linguistic POS resource available.

The UD tag set also makes principled decisions about categories that vary typologically. The AUX (auxiliary) tag separates auxiliary verbs from main verbs, a distinction that matters in many languages. ADP covers both prepositions (as in English) and postpositions (as in Japanese, Turkish, and many other languages), recognizing that the functional role is the same regardless of word order. The PART tag handles particles that do not fit elsewhere, like the English infinitive marker "to" in its verbal use, or the negative particle in languages that express negation with a separate uninflected word.

The right tag set depends on your application. Here is how to think about it:

• Use Penn Treebank tags when working on English NLP with existing tools like NLTK or spaCy in English mode, or when downstream tasks (like parsing) benefit from fine-grained distinctions
• Use Universal Dependencies tags when building multilingual systems, working with non-English languages, or when compatibility across language resources matters more than granularity
• Use custom tags only when your domain has structures neither system handles well (e.g., specialized clinical or legal notation)

Most production systems today use one of these two standards. The Penn Treebank standard dominates English-only pipelines; Universal Dependencies dominates multilingual ones.

The question of granularity is worth thinking about carefully. Finer tag sets can express more information, but they are harder to learn reliably because each individual tag sees fewer training examples. A tag set with 100 labels means that each label appears, on average, in only 1% of tokens. Rare labels may have so few training examples that the model cannot learn their distributional patterns at all. The Penn Treebank's 36 tags strike a balance that proved empirically useful: fine enough to serve syntactic parsers, coarse enough to learn with moderate-sized corpora.

The reason POS tagging is non-trivial is lexical ambiguity: most common words in English can function as more than one part of speech. The word "run" appears as VB in "I run every morning," VBZ in "She runs marathons," VBD in "He ran yesterday," VBN in "I have run before," and NN in "That was a good run." Selecting the correct tag requires examining the surrounding context.

Research on large corpora shows that roughly 40% of word types in English are ambiguous, but because ambiguous words tend to be common words (like "can," "will," "run," "light"), they account for an even higher fraction of tokens. This means a POS tagger that always picks the most frequent tag for each word will still make mistakes on a significant portion of all words it encounters.

The key statistical insight is that while individual words are ambiguous, most contexts strongly constrain the correct tag. The word "light" after "the" is almost certainly an adjective or noun, not a verb. After a pronoun like "please," it is almost certainly a verb. Context resolves most ambiguity.

The scale of lexical ambiguity in English is worth dwelling on. The 40% figure for ambiguous word types is striking, but the distribution is highly skewed. The most common 100 words in English, which account for roughly half of all tokens in typical text, include words like "will," "that," "can," "right," "even," "about," "over," "just," "back," "still," "set," "run," "leave," "play," and "watch." Almost all of these can function as multiple parts of speech. These are the words a tagger encounters most often, and they are also the hardest to tag correctly.

The difficulty compounds because ambiguous words often cluster in the same sentences. Consider "Can you leave the light on and book a table?" Every content word in this sentence is ambiguous: "can" is a modal or a noun, "leave" is a verb or an adjective, "light" is a noun or an adjective, "book" is a verb or a noun, and "table" is a noun or a verb. The correct reading requires parsing the entire sentence structure, not just examining individual tokens.

The good news for tagger developers is that although ambiguous words are frequent, the correct reading in context is usually predictable. In a large corpus of annotated English text, approximately 97% of tokens can be tagged correctly by a tagger that knows only the word identity plus the identity of the one preceding word and the one following word. The residual 3% represents hard cases where longer-range context or world knowledge is required.

Not all ambiguity is created equal. It helps to distinguish several types:

Local ambiguity: The ambiguity can be resolved by looking at immediately adjacent words. The word "can" after "I" and before "swim" is almost certainly a modal. A simple bigram model resolves this.

Non-local ambiguity: Resolution requires context further away in the sentence. "The horse raced past the barn fell" is a garden-path sentence where "raced" initially looks like the main verb but is a reduced relative clause modifier. Resolving this requires recognizing the clause structure across the whole sentence.

Structural ambiguity: The ambiguity arises from multiple ways of attaching a phrase, not just from the word's category. "I saw the man with the telescope" is ambiguous between two readings of where the prepositional phrase attaches, and this structural ambiguity affects the POS interpretation of intermediate nodes.

Annotation ambiguity: Some tokens are underdetermined even for human experts. Linguistic guidelines sometimes resolve these by convention rather than principle, which means different annotation projects make different choices and taggers trained on those projects will disagree.

Understanding these types matters practically: local ambiguity is easy to resolve with context windows, non-local and structural ambiguity requires sequence-level modeling, and annotation ambiguity sets an irreducible ceiling on tagger accuracy.

POS tagging is a supervised classification problem. Given a word and its context, predict the tag. This framing opens up the full toolkit of machine learning methods.

The challenge is that POS tagging operates at the sequence level: we are not classifying tokens independently but finding the most probable tag sequence for an entire sentence. The tags interact, because knowing that one word is a determiner strongly constrains the possible tags for the following word (determiners are almost always followed by adjectives or nouns).

The two main approaches to capturing these sequence dependencies are:

1. Generative models: Model the joint probability $P(\text{words}, \text{tags})$ and find the tags that maximize it. Hidden Markov Models (HMMs) are the classic example.
2. Discriminative models: Model $P(\text{tags} | \text{words})$ directly, conditioning on all observable information. Conditional Random Fields (CRFs) and neural sequence models take this approach.

We will cover HMMs and CRFs in depth in subsequent chapters. Here we focus on the features and evaluation principles that apply across methods.

What information helps predict a word's POS tag? Feature engineering reveals what contextual signals are informative:

Word identity: The word itself is the strongest single feature. "The" is almost always a determiner. "Quickly" is almost always an adverb.

Suffix features: Morphological endings are highly predictive. Words ending in "-tion" are almost always nouns (perception, relation). Words ending in "-ly" are almost always adverbs (quickly, slowly). Words ending in "-ing" are gerunds or present participles (running, eating).

Prefix features: Less predictive than suffixes but still useful. Words beginning with "un-" are often adjectives or verbs (unhappy, undo).

Capitalization: Capitalized words mid-sentence are likely proper nouns (NNP). Sentence-initial capitalization is uninformative but all-caps words are often abbreviations.

Context window: The tags and words of neighboring tokens. After a determiner, a noun or adjective is expected. After a modal verb like "can," a base-form verb typically follows.

Word shape: Patterns like "Xxxxx" (capitalized word), "XXXXX" (all-caps), "99.99" (decimal number) capture orthographic structure independent of the specific characters.

The relative importance of these features is not uniform across all words. For common, frequent words that appear many times in training data, word identity is by far the most useful feature. For rare or out-of-vocabulary words, morphological features (especially suffixes) become critical. A model that relies entirely on word identity will fail on unseen vocabulary; a model that ignores word identity will fail on common ambiguous words. Good feature engineering combines both.

The suffix features "umps" and "mps" might not look informative for "jumps" in isolation, but when the model sees the suffix "-s" on a word following an adjective-modified noun, it learns that this pattern almost always signals a third-person singular present verb (VBZ). Feature engineering is about giving the model the right vocabulary to express these patterns.

One limitation of feature engineering is that individual features do not capture interactions. The feature "suffix is -ed" fires for both past tense verbs (walked) and past participle adjectives (excited, concerned). The feature alone is ambiguous. But the combination "suffix is -ed AND previous word is 'have' or 'has'" almost uniquely identifies a past participle in perfect aspect. Traditional feature engineering handles this by creating conjunction features, but the number of possible conjunctions grows combinatorially with the number of base features.

This is one key motivation for neural approaches to POS tagging. Rather than manually specifying which feature combinations matter, a neural network learns to form its own internal representations that capture the relevant interactions. A bidirectional LSTM reading the sequence "has walked" will produce a hidden state for "walked" that reflects the presence of the auxiliary "has" in the left context, without any explicit feature engineering. The word embedding for "has" influences the LSTM state, which in turn influences the output distribution over tags for "walked." The model learns the interaction implicitly.

The earliest POS taggers were purely rule-based systems. Linguists hand-crafted disambiguation rules based on their knowledge of English grammar.

The most influential rule-based tagger was ENGTWOL (later called the Brill tagger's predecessor), developed at Helsinki in the 1980s. The approach was:

1. Look up each word in a lexicon to get its set of possible tags
2. Apply constraint rules that eliminate impossible tags based on context

For example, a rule might say: "If a word can be DT (determiner) or JJ (adjective) and the next word is a noun, then it is DT." These rules were written by hand by linguists with deep grammar knowledge.

Rule-based systems have a significant advantage: their decisions are interpretable. You can always trace why the tagger made a particular decision to a specific rule. They also perform well on well-formed text where grammatical rules hold cleanly.

The disadvantage is labor intensity: writing complete rule sets for English takes years of expert effort, and maintaining them as language evolves is costly. Rule-based systems also struggle with informal text where grammatical rules break down.

The ENGTWOL system relied on a two-stage architecture. In the first stage, a morphological analyzer looked up each word form in a large lexicon and returned the set of all possible morphosyntactic readings. A word like "running" might receive readings as a present participle (VERB PRES-PART), a gerund (NOUN), and an adjective. In the second stage, a Constraint Grammar formalism applied ordered rules to eliminate incompatible readings until (ideally) only one reading remained for each word. The rules expressed constraints like "a word with a determiner reading should not retain that reading if the preceding word already has a determiner reading," which captured the fact that two consecutive determiners are very rare in English.

The strength of this approach was its modularity. Rules could be added, removed, or reordered to fix specific error types without disrupting the rest of the system. Linguists maintaining the system could reason about it at a level of abstraction that matched their grammatical intuitions. When the system made an error, debugging was straightforward: find the rule that should have applied but did not, or find the rule that applied when it should not have.

The Brill tagger (1992) made a breakthrough by combining rule-based ideas with corpus learning. Instead of hand-writing rules, it learned transformation rules automatically from annotated data.

The algorithm works in two phases:

Phase 1 (Initial tagging): Assign each word its most frequent tag based on a lexicon learned from the training corpus. Words not seen in training get a default tag based on capitalization and suffix.

Phase 2 (Rule learning): Iteratively learn error-correcting transformation rules. Each rule has the form: "Change tag $A$ to tag $B$ when condition $C$ holds." The algorithm finds the rule that corrects the most errors on the training data, applies it, and repeats.

The Brill tagger typically achieves accuracy around 95-97% on the Penn Treebank test set, competitive with more complex statistical approaches of its era. Its learned rules are interpretable and can be inspected by linguists.

What made the Brill tagger historically important was the insight that the gap between statistical learning and symbolic reasoning could be bridged. The learned transformation rules looked like linguistically motivated rules: "change VBD to VBN when the previous tag is MD or HVZ" is a rule about perfect aspect that any syntactician would recognize. This interpretability distinguished Brill's approach from the black-box statistical models that were also gaining traction in the early 1990s. It showed that you could have both: rules that are learned automatically from data and that remain interpretable after learning.

The Brill tagger's main limitation is its greedy learning algorithm. At each step, it finds the single best transformation rule and applies it, then finds the next best, and so on. This greedy strategy does not guarantee finding the globally optimal rule set. Two rules might interact synergistically (rule A enables rule B to apply to more cases), but a greedy algorithm might choose a different rule A' that is locally better but prevents B from working. Dynamic programming approaches can find better solutions but at higher computational cost.

Statistical taggers learn tag assignments from annotated corpora without hand-crafted rules. They model the probability of tag sequences and select the most probable sequence.

The simplest statistical tagger assigns each word its most frequent tag from the training corpus and assigns a default tag (typically NN) to unknown words. Despite its simplicity, this baseline achieves surprisingly high accuracy: around 90-92% on standard benchmarks. Most words in natural text are unambiguous in context, and even for ambiguous words, the most frequent reading is often correct.

This baseline is important for calibration. Any more sophisticated method must substantially beat 90%+ accuracy to justify its complexity.

The 90%+ baseline accuracy comes largely from function words, which are extremely common and almost never ambiguous. "The" is almost always DT; "of" is almost always ADP. The hard cases are the content words with multiple possible tags.

Better taggers model the entire tag sequence jointly. The key idea is to find the tag sequence $\mathbf{t} = (t_1, t_2, \ldots, t_n)$ that maximizes the probability given the word sequence $\mathbf{w} = (w_1, w_2, \ldots, w_n)$:

This formulation treats tagging as searching for the globally optimal sequence, not assigning each token independently. The key question is how to model $P(\mathbf{t} \mid \mathbf{w})$ efficiently without enumerating all possible tag sequences (which grows exponentially with sentence length).

Hidden Markov Models (covered in the next chapter) approach this by decomposing the joint probability $P(\mathbf{w}, \mathbf{t})$ using the Markov assumption: each tag depends only on the previous tag (the transition probability) and each word depends only on its tag (the emission probability). This decomposition allows efficient computation via dynamic programming.

where:

• $P(t_i \mid t_{i-1})$: the probability of transitioning from tag $t_{i-1}$ to tag $t_i$ (transition probability)
• $P(w_i \mid t_i)$: the probability of generating word $w_i$ given tag $t_i$ (emission probability)

Both probability tables are estimated from counts in the annotated training corpus.

To understand why this decomposition is useful, consider the naive alternative. Without any structural assumption, $P(\mathbf{t} \mid \mathbf{w})$ is a distribution over all possible tag sequences of length $n$. If there are $K$ possible tags, there are $K^n$ possible sequences for a sentence of length $n$. For English with 36 Penn Treebank tags and a sentence of 20 words, that is $36^{20} \approx 10^{31}$ sequences to compare. The Markov assumption reduces this to $O(K^2 \cdot n)$ computations using the Viterbi algorithm, which is polynomial rather than exponential.

The Markov assumption is a simplification, of course. In reality, the correct tag for a word might depend on context three or four words away. Trigram HMMs (where each tag depends on the two previous tags) capture more context and generally outperform bigram HMMs. But even trigram models eventually hit the limits of local context modeling, and the jump to CRFs and neural models comes from allowing conditioning on the full input sequence.

Let us build intuition for these two probability types by computing them from real data.

The transition table reveals clear linguistic patterns. After DET (determiner), NOUN gets a high probability because determiners introduce noun phrases. After VERB, ADP (adposition/preposition) is likely because verbs are often followed by prepositional phrases. These patterns are exactly what a syntactic parser exploits, but here they emerge automatically from corpus statistics.

The emission probabilities tell a complementary story. A word like "the" has very high emission probability under the DET tag and near-zero probability under any other tag. A word like "run" has moderate emission probability under both VERB and NOUN. This reflects its ambiguity. The ratio of these emission probabilities is what the HMM uses to decide which tag is more likely in context: if the transition probabilities favor VERB in a given position, and "run" has moderate emission probability under VERB, then VERB wins.

Laplace smoothing (adding 1 to all counts) prevents zero probabilities for unseen word-tag combinations. Without smoothing, any word not seen in training with a given tag gets an emission probability of exactly 0, which would prevent the model from ever assigning that tag to that word. Smoothing distributes a small amount of probability mass to unseen combinations, allowing the model to handle novel words by falling back on the tag distribution implied by the transition probabilities.

Beyond HMMs, discriminative classifiers can be applied in a "greedy left-to-right" mode: tag each word using the current word's features plus the predicted tag of the previous word. This is not globally optimal like Viterbi decoding, but it allows rich, overlapping feature sets.

The improvement from adding rich features and the previous predicted tag demonstrates the value of contextual information. Even a simple greedy left-to-right decoding with logistic regression substantially outperforms the most-frequent-tag baseline.

The greedy decoding strategy introduces a potential problem: errors compound. If the tagger predicts the wrong tag for position $i$, it uses that wrong tag as a feature for position $i+1$, which can cause a cascade of errors. For most sentences, this compounding effect is mild because the model can recover from a single-token error once sufficient new context is available. But in pathological cases, especially at the start of ambiguous sentences, a single wrong prediction can derail the entire sequence.

This is one of the key advantages of global decoding methods like the Viterbi algorithm used in HMMs and CRFs. Those methods compute the globally optimal sequence rather than making locally greedy decisions, so a single locally bad choice does not necessarily propagate. The tradeoff is computational: greedy decoding is $O(n \cdot K)$ while Viterbi is $O(n \cdot K^2)$, where $K$ is the number of tags. For small tag sets (like the 17 UD tags or 36 PTB tags), Viterbi is still very fast in practice.

Production POS tagging today uses neural models. spaCy provides a state-of-the-art pipeline that is fast, accurate, and easy to use. Let us see it in action on real text:

Notice how spaCy correctly disambiguates "bank" (NOUN) from a financial verb sense, correctly tags both "book" instances as VERB in the third sentence, and resolves "will" as a modal (AUX) in context. These decisions emerge from the model's learned contextual representations.

The .pos_ attribute gives the Universal Dependencies UPOS tag; .tag_ gives the Penn Treebank fine-grained tag. Both are available simultaneously.

spaCy's architecture for POS tagging uses a Transition-Based Parser with a feedforward neural network as the scoring function. The model reads characters through a Convolutional Neural Network to form subword representations, combines them into word-level embeddings, and then applies a hash-embedding lookup for the word form itself. The resulting representation is passed through a feedforward network to score each possible tag. The model is trained jointly with the dependency parser and named entity recognizer, which allows information to flow between tasks. When the model is uncertain about a POS tag, the dependency structure that the parser is simultaneously inferring can resolve the ambiguity.

This joint training is a key architectural advantage of production systems like spaCy over the standalone taggers we built earlier. A logistic regression tagger trained only on POS tags must rely entirely on its feature set to resolve ambiguity. A jointly trained system can use the parser.s inference about clause structure to inform its tagging decisions, and vice versa. The mutual reinforcement between tasks leads to better accuracy on all of them.

The standard metric for POS tagging is per-token accuracy: the fraction of tokens in the test set that receive the correct tag.

This seems simple, but the metric has important nuances:

Known-word vs. unknown-word accuracy: A word seen in training may be tagged reliably; a word never seen in training must be handled by generalizing from morphology and context. Reporting accuracy separately for known and unknown words reveals where a tagger's weaknesses lie.

Ambiguous vs. unambiguous tokens: Tagging "the" correctly is trivial. Tagging "run" correctly after a noun requires context modeling. Accuracy on ambiguous tokens is a much more diagnostic metric.

Domain sensitivity: A tagger trained on Wall Street Journal articles will degrade on Twitter or biomedical text. Domain-specific accuracy matters for real applications.

The gap between known and unknown word accuracy is diagnostic. A large gap means the tagger memorizes training data rather than generalizing. Good taggers use morphological features (suffixes, prefixes, capitalization) to make reasonable guesses about unseen words.

Per-token accuracy hides which tag pairs are most often confused. A confusion matrix reveals the systematic errors:

The diagonal represents correct predictions. Off-diagonal entries represent confusions. The most common errors in English POS tagging are:

• NOUN vs VERB: Words like "run," "book," and "call" can be both
• ADJ vs ADV: Words like "fast," "hard," and "long" function as both
• ADP vs SCONJ: "after," "before," and "since" can be prepositions or subordinating conjunctions

These confusions are exactly where context matters most, and they represent the hard residual error that even state-of-the-art systems struggle with.

For some applications, accuracy on a specific tag matters more than overall accuracy. A system building a verb phrase extractor cares most about verb recall. We can compute per-tag precision and recall:

Tags with high support (many training examples) generally achieve better scores. Rare tags like INTJ or SYM have fewer examples and often lower accuracy. This uneven performance is a practical concern: if your application depends on tagging a specific grammatical category, check that category's performance, rather than relying on the overall accuracy alone.

The per-tag breakdown also reveals the asymmetry between precision and recall for different tags. High-precision, low-recall means the tagger is conservative: when it assigns that tag, it is usually right, but it misses many true instances. High-recall, low-precision means it is aggressive: it catches most true instances but also fires on many false positives. This distinction matters for downstream use. A system that uses PROPN tags to find named entity candidates needs high recall (do not miss real proper nouns) more than high precision (false positive proper nouns can be filtered later).

One often-overlooked aspect of POS tagger evaluation is that the "gold" labels in test sets are themselves imperfect. Human annotators applying linguistic guidelines to complex sentences sometimes disagree. Studies of POS annotation projects typically find inter-annotator agreement of 97-98% on standard English text. This means that even a perfect tagger (one that perfectly captures the consensus judgment of a typical annotator) would achieve at most 97-98% accuracy on a test set annotated by different annotators.

The practical implication is that when you see a POS tagger reporting 98% accuracy, you should interpret this carefully. Some of that remaining 2% error rate is tagger error. Some of it is annotation inconsistency: the tagger made a linguistically defensible choice that happened to differ from the one made by the test set annotators. Distinguishing these two sources of error requires human review of the errors, not just aggregate metrics.

This ceiling becomes important when comparing systems that are all performing above 97% accuracy. The differences between modern neural systems at this accuracy level are often smaller than the annotation noise in the test set. Reporting confidence intervals around accuracy estimates is good practice, but unfortunately still rare in NLP publications.

POS tags are rarely the end goal. They serve as features or intermediate representations for richer NLP tasks.

Named entity recognizers benefit from POS tags because entity names follow characteristic POS patterns. A person's name is typically NNP NNP (proper noun proper noun). An organization name often ends with NNP NNP NNP. These patterns help the NER system focus its attention:

This heuristic is not perfect (it will miss multi-word entities that include non-proper-nouns), but it shows how POS tags provide a cheap first approximation for entity detection.

POS tags enable pattern-based information extraction. Noun chunks (sequences of determiners, adjectives, and nouns) identify the subjects and objects of sentences. Verb phrases identify actions. This structured extraction is faster and more interpretable than neural extraction for well-defined patterns:

POS tags are essential for correct lemmatization. The word "saw" has two lemmas depending on its POS: as a verb (VBD), it lemmatizes to "see"; as a noun (NN), it lemmatizes to "saw" (a cutting tool). Without POS information, a lemmatizer must guess:

This dependency on POS for correct lemmatization propagates through the entire NLP pipeline. A wrong POS tag early in the pipeline causes wrong lemmatization, which causes wrong feature matching downstream.

One of the most direct uses of POS tags is shallow parsing, also known as chunking. A chunker groups consecutive tokens into non-overlapping phrases, identifying noun phrases, verb phrases, and prepositional phrases without building a full recursive parse tree.

The most common chunking task is noun phrase chunking (also called base noun phrase chunking or NP chunking). A base NP is a simple, non-recursive noun phrase: the outermost noun phrase that does not itself contain any other noun phrases. For example, in "the quick brown fox jumps over the lazy dog," the base NPs are "the quick brown fox" and "the lazy dog." The verb phrase "jumps over" and the prepositional phrase "over the lazy dog" are not base NPs.

POS tags provide the necessary information for NP chunking. The pattern for an English base NP is roughly: an optional determiner, zero or more adjectives, and one or more nouns. In Penn Treebank tags, this pattern is DT? JJ* NN+ (where ? means zero or one, * means zero or more, and + means one or more). A simple regex over the tag sequence captures this structure:

Chunking with POS patterns is fast and requires no additional training data beyond the POS tagger itself. The resulting noun phrases can be used directly as features for downstream tasks. In information extraction, noun phrase chunks are candidate argument slots for relation extraction. In text summarization, high-frequency noun phrases identify the key entities being discussed. In search engine indexing, noun phrase chunking extracts multi-word key terms that are more informative than individual words.

The limitation of pattern-based chunking is that it handles only the simplest phrase structures. Complex noun phrases containing relative clauses ("the company that won the contract"), appositive phrases ("Apple Inc., the Cupertino-based technology giant"), or coordination ("cats and dogs") require recursive parsing or more sophisticated models.

The transition from statistical to neural approaches for POS tagging followed the broader trajectory of NLP. BiLSTM-based models appeared around 2015-2016 and substantially improved accuracy, particularly on unknown words. Transformer-based models (starting with BERT in 2018) pushed accuracy even higher by using large-scale pre-training on unlabeled text.

A bidirectional LSTM (BiLSTM) tagger reads the input sentence in both forward and backward directions, producing a contextual representation for each token that depends on the entire sequence, rather than only a local window. The forward LSTM reads left-to-right, building up a hidden state that summarizes everything seen so far. The backward LSTM reads right-to-left, building a hidden state that summarizes everything from the right. Concatenating these two states produces a representation that is sensitive to context from both directions.

This bidirectionality is particularly valuable for POS tagging because POS ambiguity is often resolvable only with both left and right context. "I saw him" and "I saw a" both start with "I saw," but "him" (a pronoun) is a strong signal that "saw" is a past-tense verb, while "a" (a determiner introducing a noun phrase) is also consistent with that reading. However, "She picked up the saw" requires right context to confirm that "saw" is a noun: the next word "She" suggests a new clause was just completed, and picking something up implies a physical object.

The BiLSTM approach to POS tagging operates as follows. Each input word is first converted to a dense vector embedding (either a lookup embedding or a character-level encoding). The BiLSTM reads the sequence of embeddings and produces a contextual representation for each position. This representation is then passed through a linear layer followed by a softmax to produce a probability distribution over tags. The model is trained end-to-end with cross-entropy loss to predict the correct tag at each position.

Character-level encoding is important for handling unknown words. Rather than looking up a word in an embedding table (which fails for out-of-vocabulary words), a character-level CNN or LSTM reads the character sequence of the word and produces a representation that captures morphological patterns. A BiLSTM tagger that combines word embeddings with character-level encodings achieves much better unknown-word accuracy than one that uses only word embeddings.

BERT (Bidirectional Encoder Representations from Transformers) changed POS tagging in the same way it changed every other NLP task: by providing contextual word representations of unprecedented quality, pre-trained on billions of words of text. Fine-tuning BERT on a POS-annotated dataset requires only a simple linear classification head on top of BERT's output representations.

The key advantage of BERT over BiLSTMs for POS tagging is the quality of its contextual representations. BERT was pre-trained to predict masked words in context, which forced it to learn deep syntactic and semantic patterns. Its representations encode the word's distributional properties and the way its meaning and syntactic role shift across contexts. When you ask a BERT-based tagger to tag "run" in "She goes for a morning run," the BERT representation of "run" already reflects the fact that it follows a determiner-adjective-noun sequence, making it much easier for the classification head to assign NOUN.

The practical workflow for BERT-based POS tagging uses the Hugging Face transformers library. The input sentence is tokenized using BERT's subword tokenizer, the subword tokens are passed through BERT to get contextual embeddings, and then a linear layer maps the embedding at each word position to a distribution over POS tags. One complication is that BERT's subword tokenization splits words into pieces (e.g., "running" might become "run" + "##ning"), so the tag assignment needs to be mapped back from subword positions to whole-word positions.

Here is a simplified illustration of what that workflow looks like:

The alignment strategy is to predict the tag from the first subword token of each word and ignore subsequent subword pieces during loss computation. This works well in practice because the contextual representation of the first subword piece already incorporates information from the entire word (through BERT's self-attention mechanism).

Current POS tagging accuracy on the Penn Treebank Wall Street Journal test set:

The state-of-the-art accuracy around 98-99% sounds impressive, but consider that an error rate of 2% across a 30-word sentence means roughly a 45% chance of at least one error per sentence. For tasks where every token must be correctly tagged (like precise syntactic parsing), this matters.

The gains from simple baselines to transformer models come from:

• HMMs: Sequence modeling, transition constraints
• MaxEnt: Rich contextual features, overlapping features possible
• BiLSTM: Bidirectional context, learned representations
• BERT: Pre-trained on massive text, contextual embeddings that capture syntax implicitly

The diminishing returns above 97% reflect the fundamental ambiguity ceiling: for some tokens, the correct tag is underdetermined from the local context, and human annotators themselves sometimes disagree.

Understanding the distribution of POS tags in a corpus reveals the characteristics of different text genres:

The comparison plot makes the tradeoff clear. The most-frequent-tag baseline handles known words almost as well as the MaxEnt tagger because most known words are unambiguous. The real gains come on unknown words, where morphological features and context allow the MaxEnt tagger to generalize. The raw spaCy result is much lower in this illustrative evaluation because spaCy applies its own tokenization and the code aligns predictions to Brown tokens by position after joining each sentence. Domain and annotation differences add further mismatch. Treat that bar as a practical warning about evaluation alignment, not as evidence that neural taggers are generally less accurate.

One of the most active research areas in POS tagging is cross-lingual transfer: training a model on one language and applying it to another, with little or no annotated data in the target language. This problem is important because the world has thousands of languages, and creating annotated training data for all of them is infeasible.

The Universal Dependencies project provides the infrastructure for cross-lingual POS work by standardizing both the tag set and the annotation format across languages. A model trained on English UD data uses the same 17 UPOS tags as a model trained on Spanish, Russian, or Swahili data. This shared label space makes it possible, in principle, to transfer a model trained on a high-resource language to a low-resource language.

The challenge is that even with a shared label space, languages differ in their surface forms, morphology, and word order. A model trained on English encounters word sequences like DET ADJ NOUN. Japanese uses postpositions instead of prepositions, and the verb comes last rather than in the middle. A model that learns English distributional patterns cannot directly apply them to Japanese.

Multilingual pre-trained models like mBERT (multilingual BERT) and XLM-RoBERTa have made cross-lingual transfer much more effective. These models are pre-trained on text in over 100 languages simultaneously, learning representations that are partially language-agnostic. When you fine-tune mBERT on English POS annotations and then evaluate on, say, Bulgarian, the model achieves surprisingly high accuracy despite having seen no Bulgarian POS annotations. The pre-training has learned to align representations across languages for related concepts, and the fine-tuning in English teaches the classification head what the UPOS labels mean.

The degree of transfer depends heavily on linguistic relatedness. Transfer from English to German works well (both are Germanic languages with similar morphological and syntactic patterns). Transfer from English to Turkish is harder because Turkish is an agglutinative language with very different morphological structure and verb-final word order. Transfer from English to Chinese is somewhere in between: Chinese has minimal morphology (which simplifies some problems) but very different syntactic constructions.

Working with POS taggers in real applications involves several practical considerations that rarely appear in academic papers but matter enormously for deployed systems.

POS taggers are sensitive to tokenization. A tagger trained on Penn Treebank tokenization expects "I'm" to be split into "I" and "'m," "don't" into "do" and "n't," and "New York" to remain a two-token sequence. If your preprocessing uses a different tokenization scheme, the tagger may produce wrong results even if it is otherwise accurate.

The safest approach is to use the same tokenizer that was used to train the tagger. spaCy's tagger and tokenizer are designed as a unit: the tokenizer produces tokens in the format that the tagger expects. If you feed pre-tokenized input to spaCy's tagger, you can use doc = nlp.make_doc(text) followed by nlp.get_pipe("tagger")(doc) to bypass spaCy's tokenizer.

A tagger trained on news text will degrade on social media, where abbreviations, misspellings, emojis, and non-standard grammar are common. "lol" does not appear in the Penn Treebank, so the tagger has no word-identity feature for it. The fallback to morphological features and capitalization heuristics may assign it NOUN (the most common default) rather than the more accurate INTJ. For social media applications, retraining or fine-tuning on domain-specific annotated data improves performance substantially.

In production systems, you need to make a deliberate choice about the speed-accuracy tradeoff. The options roughly are:

• Rule-based taggers: Fastest (millions of tokens per second), lowest accuracy (95-96%)
• Statistical taggers (HMM, MaxEnt): Fast (hundreds of thousands of tokens per second), good accuracy (97%)
• Neural taggers (BiLSTM): Moderate speed (tens of thousands of tokens per second), high accuracy (98%)
• Transformer taggers (BERT): Slow (thousands of tokens per second on CPU), highest accuracy (98-99%)

For a system processing large volumes of text offline, a transformer tagger may be acceptable. For a real-time application requiring sub-100ms latency, a statistical or BiLSTM tagger is more practical. Batch processing with GPU acceleration can make transformer taggers much faster (hundreds of thousands of tokens per second on a modern GPU), but requires GPU infrastructure.

In a classical NLP pipeline, POS errors propagate to downstream components. An incorrect verb tag may cause the chunker to create a wrong chunk boundary, which may cause the dependency parser to attach an argument to the wrong head, which may cause the NER to miss an entity boundary. This cascade is one of the motivations for joint or end-to-end models that learn to perform multiple tasks simultaneously, sharing information across tasks to reduce the compounding of errors.

In practice, the cascading error problem is less severe than it sounds, because downstream models are usually trained to tolerate noisy POS input. A neural NER model trained with noisy predicted POS features will learn to down-weight those features when they are unreliable. But for high-stakes applications where every error matters, understanding the error cascade and testing each pipeline stage separately is important.

POS tagging has matured to the point where on clean, formal English text it approaches human-level accuracy. But several practical limitations remain important for anyone applying these systems.

Domain adaptation is one of the most significant challenges. A tagger trained on Wall Street Journal news articles degrades substantially when applied to Twitter posts, biomedical literature, legal text, or programming code. The vocabulary shifts, syntax is different, and even the tag distributions change. Financial text has more cardinal numbers; biomedical text has more complex noun compounds; social media text has more interjections and non-standard forms. The standard practice of fine-tuning a general-purpose model on domain-specific annotated data usually helps, but collecting that data takes effort.

Language coverage presents another challenge. The resources available for English (multiple large treebanks, pre-trained models, evaluation benchmarks) are exceptional. For many of the world's languages, annotated data is scarce, pre-trained models are of lower quality, and even basic tag set decisions are contested. Universal Dependencies has expanded coverage significantly, but the quality gap between English and low-resource languages remains large.

Error propagation through pipelines is a subtler issue that practitioners sometimes underestimate. A POS tagger error at 2% per token means roughly a 45% chance of at least one error in a 30-word sentence. If a downstream parser, NER system, or relation extractor depends on those tags being correct, those errors propagate and multiply. Pipeline errors compound. This is one reason modern end-to-end neural systems often bypass explicit POS tagging and learn to model syntax implicitly, but they give up interpretability in return.

Speed is a practical constraint in production. Transformer-based taggers achieve near-human accuracy but are slow. For a system tagging millions of documents per day, the difference between a BiLSTM tagger running at 50,000 tokens per second and a BERT-based tagger running at 1,000 tokens per second matters enormously. The choice of tagger involves a speed-accuracy tradeoff that depends on your throughput requirements.

Finally, the ceiling imposed by annotation disagreement is real but often underappreciated. Linguists disagree on the correct POS tag for ambiguous tokens. The Penn Treebank annotation guidelines resolved many cases by fiat, but those choices do not always reflect principled linguistic reasoning. A tagger cannot do better than the consistency of its training data, and some residual error cannot be resolved.

The question of whether explicit POS tagging will remain relevant in the era of large language models raises a practical question. Models like GPT-4 and Claude clearly understand syntax implicitly; they do not need explicit POS tags to parse sentences or generate grammatically correct text. But for applications requiring interpretable, structured output, where you need to know the model's answer and why it produced that answer, POS tags remain valuable. They provide a compact, human-readable summary of syntactic structure that can be audited, corrected, and used as a feature in systems that require transparency.

Part-of-speech tagging assigns a grammatical category label to each token in a sequence. The key ideas from this chapter:

• Tag sets define the label space: The Penn Treebank set (36 tags) is the standard for English; Universal Dependencies (17 tags) enables cross-linguistic work. The choice involves a tradeoff between granularity and generality.
• Ambiguity is the core challenge: Common words can appear as multiple parts of speech. Resolving this ambiguity requires contextual information, not just word identity. Roughly 40% of English word types are ambiguous, and those ambiguous words account for a disproportionate share of tokens.
• Features drive accuracy: Word identity, morphological suffixes, capitalization, neighboring words, and previous predicted tags are the key features. Unknown words require morphological generalization; known words rely heavily on word identity.
• Sequence modeling beats token-by-token classification: HMMs, CRFs, and neural sequence models exploit tag-to-tag dependencies that independent classifiers ignore. The globally optimal sequence found by Viterbi decoding is typically better than greedy left-to-right decoding.
• Accuracy benchmarks have a ceiling: State-of-the-art systems reach 98-99% on standard benchmarks, but the remaining 1-2% includes ambiguous tokens and annotation noise. Inter-annotator agreement itself is only 97-98%, setting a practical upper bound.
• POS tags serve downstream tasks: Named entity recognition, syntactic parsing, information extraction, chunking, and lemmatization all benefit from POS annotations. Errors in POS tagging propagate through downstream components.
• Domain matters: Taggers trained on news text degrade on social media, biomedical, or legal text. Evaluation should always include domain-matched test data.
• Cross-lingual transfer is possible: Multilingual pre-trained models enable POS tagging for low-resource languages by transferring representations from high-resource languages through a shared label space.

In the next chapter, we will see how POS tagging extends to the more complex problem of Named Entity Recognition, where the labels capture grammatical role and semantic entity type.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about part-of-speech tagging.