Whole Word Masking › Visualizing the Difference

Michael BrenndoerferUpdated July 10, 20253 min read
Language AI HandbookAll chapters
Part 22 of 67Pre-training Objectives
Chapter 157Whole Word Masking › Visualizing the Difference
Reading section 6 of 13 · Visualizing the Difference · 3 minSection 6 of 13 · 3 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 157 of 513

Section 6 of 13 · 3 min read

Whole Word Masking › Visualizing the DifferenceOne section · Sections in this chapter6 / 13 ↓
Sections in this chapter13
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let's create a visualization comparing how both masking strategies affect a set of sentences with multi-token words. We use a higher masking probability (30%) to make the patterns more visible in the diagram.

Out[11]:
Visualization
Grid showing sentences with color-coded tokens comparing masking strategies. This shows WWM keeps word boundaries intact.
Comparison of random subword masking vs whole word masking on three sample sentences containing multi-token words. Green tokens are visible, red tokens are masked with [M]. Random subword masking produces fragmented patterns where word roots are visible while suffixes are masked (information leakage). Whole word masking consistently hides entire words, creating a genuinely harder prediction task.

The visualization highlights how random subword masking creates fragmented masking patterns that leak information. In "transformation", random masking might leave "transform" visible while masking "##ation", handing the model a near-trivial suffix prediction task. In "internationalization", which tokenizes into many fragments, random masking might mask only a middle fragment while leaving both prefix and suffix visible, making prediction even easier. WWM eliminates this entirely by treating words as atomic units: all fragments of a selected word are masked simultaneously, or none are.