Part-of-Speech Tagging › Visualizing Tag Distributions

Michael BrenndoerferApril 16, 20255 min read
Language AI HandbookAll chapters
Part 6 of 67Sequence Labeling
Chapter 36Part-of-Speech Tagging › Visualizing Tag Distributions
Reading section 13 of 18 · Visualizing Tag Distributions · 5 minSection 13 of 18 · 5 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 36 of 513

Section 13 of 18 · 5 min read

Part-of-Speech Tagging › Visualizing Tag DistributionsOne section · Sections in this chapter13 / 18 ↓
Sections in this chapter18
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Understanding the distribution of POS tags in a corpus reveals the characteristics of different text genres:

Out[37]:
Visualization
Bar chart showing frequency of each POS tag in descending order, with NOUN and VERB as the most common.
POS tag frequency distribution in the Brown corpus training subset (universal tagset). NOUN and VERB dominate content-bearing positions, while DET and ADP are high-frequency function words. The long tail of infrequent tags (INTJ, SYM, X) reflects the rare but linguistically important categories the tag set must cover.
Out[38]:
Visualization
Grouped bar chart comparing POS tagging accuracy for three methods split by known vs unknown words.
POS tagging accuracy by method and word type (known vs. unknown words) on the Brown corpus test subset. The most-frequent-tag baseline and MaxEnt tagger perform similarly on known words, while the MaxEnt tagger achieves substantially higher accuracy on unknown words by leveraging morphological features. The much lower raw spaCy scores reflect this illustrative comparison's tokenization, positional alignment, and domain mismatch, so they should not be read as a general neural benchmark.

The comparison plot makes the tradeoff clear. The most-frequent-tag baseline handles known words almost as well as the MaxEnt tagger because most known words are unambiguous. The real gains come on unknown words, where morphological features and context allow the MaxEnt tagger to generalize. The raw spaCy result is much lower in this illustrative evaluation because spaCy applies its own tokenization and the code aligns predictions to Brown tokens by position after joining each sentence. Domain and annotation differences add further mismatch. Treat that bar as a practical warning about evaluation alignment, not as evidence that neural taggers are generally less accurate.