Part-of-Speech Tagging › Evaluating POS Taggers
Language AI HandbookAll chapters Part-of-Speech Tagging › Evaluating POS TaggersOne section · Sections in this chapter9 / 18 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Per-Token Accuracy
The standard metric for POS tagging is per-token accuracy: the fraction of tokens in the test set that receive the correct tag.
This seems simple, but the metric has important nuances:
Known-word vs. unknown-word accuracy: A word seen in training may be tagged reliably; a word never seen in training must be handled by generalizing from morphology and context. Reporting accuracy separately for known and unknown words reveals where a tagger's weaknesses lie.
Ambiguous vs. unambiguous tokens: Tagging "the" correctly is trivial. Tagging "run" correctly after a noun requires context modeling. Accuracy on ambiguous tokens is a much more diagnostic metric.
Domain sensitivity: A tagger trained on Wall Street Journal articles will degrade on Twitter or biomedical text. Domain-specific accuracy matters for real applications.
# Compare known-word vs unknown-word accuracy for our MaxEnt tagger
train_vocab = set(word.lower() for sent in train_sents for word, _ in sent)
correct_known = total_known = 0
correct_unknown = total_unknown = 0
for sent in test_sents:
words = [w for w, _ in sent]
gold = [t for _, t in sent]
pred = [t for _, t in tag_sentence_greedy(words, clf, vec, le)]
for w, g, p in zip(words, gold, pred):
if w.lower() in train_vocab:
total_known += 1
if g == p:
correct_known += 1
else:
total_unknown += 1
if g == p:
correct_unknown += 1Accuracy by Word Type ======================================== Known words: 0.961 (15945/16598 tokens) Unknown words: 0.846 (1850/2188 tokens) Accuracy gap: 11.5 percentage points The larger the gap, the more the tagger relies on memorized word-tag associations rather than contextual and morphological generalization.
The gap between known and unknown word accuracy is diagnostic. A large gap means the tagger memorizes training data rather than generalizing. Good taggers use morphological features (suffixes, prefixes, capitalization) to make reasonable guesses about unseen words.
Confusion Matrix Analysis
Per-token accuracy hides which tag pairs are most often confused. A confusion matrix reveals the systematic errors:
The diagonal represents correct predictions. Off-diagonal entries represent confusions. The most common errors in English POS tagging are:
- NOUN vs VERB: Words like "run," "book," and "call" can be both
- ADJ vs ADV: Words like "fast," "hard," and "long" function as both
- ADP vs SCONJ: "after," "before," and "since" can be prepositions or subordinating conjunctions
These confusions are exactly where context matters most, and they represent the hard residual error that even state-of-the-art systems struggle with.
Tag-Level Precision and Recall
For some applications, accuracy on a specific tag matters more than overall accuracy. A system building a verb phrase extractor cares most about verb recall. We can compute per-tag precision and recall:
from sklearn.metrics import classification_report
# Per-tag classification report
report = classification_report(
all_gold,
all_pred,
labels=tags_present,
target_names=tags_present,
output_dict=True,
zero_division=0,
)Per-Tag Precision, Recall, F1 -------------------------------------------------- Tag Precision Recall F1 Support -------------------------------------------------- . 1.000 1.000 1.000 2115 ADJ 0.880 0.799 0.837 1560 ADP 0.955 0.970 0.962 2366 ADV 0.906 0.827 0.865 765 CONJ 0.996 0.987 0.992 552 DET 0.995 0.991 0.993 2355 NOUN 0.915 0.964 0.939 5098 NUM 0.976 0.957 0.966 345 PRON 0.996 0.934 0.964 485 PRT 0.909 0.880 0.894 475 VERB 0.952 0.954 0.953 2624 X 1.000 0.196 0.327 46 -------------------------------------------------- W. Avg 0.947 0.947 0.946 18786
Tags with high support (many training examples) generally achieve better scores. Rare tags like INTJ or SYM have fewer examples and often lower accuracy. This uneven performance is a practical concern: if your application depends on tagging a specific grammatical category, check that category's performance, rather than relying on the overall accuracy alone.
The per-tag breakdown also reveals the asymmetry between precision and recall for different tags. High-precision, low-recall means the tagger is conservative: when it assigns that tag, it is usually right, but it misses many true instances. High-recall, low-precision means it is aggressive: it catches most true instances but also fires on many false positives. This distinction matters for downstream use. A system that uses PROPN tags to find named entity candidates needs high recall (do not miss real proper nouns) more than high precision (false positive proper nouns can be filtered later).
Inter-Annotator Agreement and the Accuracy Ceiling
One often-overlooked aspect of POS tagger evaluation is that the "gold" labels in test sets are themselves imperfect. Human annotators applying linguistic guidelines to complex sentences sometimes disagree. Studies of POS annotation projects typically find inter-annotator agreement of 97-98% on standard English text. This means that even a perfect tagger (one that perfectly captures the consensus judgment of a typical annotator) would achieve at most 97-98% accuracy on a test set annotated by different annotators.
The practical implication is that when you see a POS tagger reporting 98% accuracy, you should interpret this carefully. Some of that remaining 2% error rate is tagger error. Some of it is annotation inconsistency: the tagger made a linguistically defensible choice that happened to differ from the one made by the test set annotators. Distinguishing these two sources of error requires human review of the errors, not just aggregate metrics.
This ceiling becomes important when comparing systems that are all performing above 97% accuracy. The differences between modern neural systems at this accuracy level are often smaller than the annotation noise in the test set. Reporting confidence intervals around accuracy estimates is good practice, but unfortunately still rare in NLP publications.