WordPiece › Visualizations

Michael BrenndoerferUpdated March 3, 20265 min read
Language AI HandbookAll chapters
Part 5 of 67Subword Tokenization
Chapter 30WordPiece › Visualizations
Reading section 10 of 13 · Visualizations · 5 minSection 10 of 13 · 5 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 30 of 513

Section 10 of 13 · 5 min read

WordPiece › VisualizationsOne section · Sections in this chapter10 / 13 ↓
Sections in this chapter13
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let us visualize two key aspects of WordPiece: how the merge scores differ from raw frequency counts, and how tokenization produces different subword splits across word families.

Out[16]:
Visualization
Scatter plot comparing raw pair frequency and WordPiece PMI score for token pairs, showing divergence between frequency and PMI rankings.
Comparison of raw pair frequency versus WordPiece PMI-based merge scores for candidate token pairs. The PMI score normalizes by individual token probabilities, so it ranks lower-frequency but more tightly associated pairs above highly frequent but informationally redundant pairs. Pairs above the diagonal benefit most from the PMI normalization, revealing that tightly coupled morphological units (like the ##re ##construct pair) can outscore high-frequency common pairs (like a ##nd) when co-occurrence is genuinely informative.
Out[17]:
Visualization
Grid of word tokenizations showing colored blocks for each subword token with ## continuation markers shown distinctly.
WordPiece tokenization of related word families derived from the roots ''play'' and ''construct''. Words are segmented into subword tokens, with each token shown as a colored block. Continuation tokens (prefixed with ##) appear in orange, while initial tokens appear in blue. Words in the same family share subword tokens (for example, all ''play''-family words contain the blue ''play'' block), showing how shared subwords let models reuse information across morphological variants.

The word family visualization reveals a key property of WordPiece: morphologically related words share subword tokens. The "play" family all contain the token play, and the "construct" family all share ##struct. This shared subword structure is precisely what allows BERT to generalize across word forms. When the model encounters "deconstructed", it can draw on what it has learned about de, ##con, ##struct, and ##ed independently. Each subword piece carries information learned from all the words in which it appeared, allowing BERT to compose meaning from parts even for words it has never seen as wholes.