Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Whole Word Masking › Worked Example: Step-by-Step Trace

Michael BrenndoerferUpdated July 10, 20253 min read

Language AI HandbookAll chapters

Part 22 of 67Pre-training Objectives

Chapter 157Whole Word Masking › Worked Example: Step-by-Step Trace

Reading section 4 of 13 · Worked Example: Step-by-Step Trace · 3 minSection 4 of 13 · 3 min

Section 4 of 13 · 3 min read

Whole Word Masking › Worked Example: Step-by-Step TraceOne section · Sections in this chapter4 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let's walk through a complete WWM application on a concrete sentence to solidify the algorithm. We will use the sentence "The transformation was undeniably remarkable." and trace every step by hand.

Step 1: Tokenize

First, WordPiece tokenizes the sentence:

Input:  "The transformation was undeniably remarkable."
Tokens: ["The", "transform", "##ation", "was", "undeni", "##ably", "remark", "##able", "."]

We have 9 tokens. The 15% target means we want to mask approximately $0.15 \times 9 \approx 1.35$ tokens. Since we operate at the word level, we will aim for 1-2 tokens of masking.

Step 2: Identify word groups

Scanning for ## prefixes, we group tokens into words:

Word 1: ["The"]                    (index 0, length 1)
Word 2: ["transform", "##ation"]   (indices 1-2, length 2)
Word 3: ["was"]                    (index 3, length 1)
Word 4: ["undeni", "##ably"]       (indices 4-5, length 2)
Word 5: ["remark", "##able"]       (indices 6-7, length 2)
Word 6: ["."]                      (index 8, length 1)

Six words total, with lengths [1, 2, 1, 2, 2, 1]. Total tokens: 9.

Step 3: Shuffle and greedily select words

Suppose the random shuffle produces the order: [Word 4, Word 1, Word 5, Word 2, Word 3, Word 6].

We step through this order, accumulating token count until we reach approximately 15% (about 1-2 tokens):

Consider Word 4 ("undeni", "##ably"): total masked would be 2 tokens (22%). This is close to our target and within acceptable variance, so we select it and stop.

The selected word is Word 4. This gives us a masking ratio of 2/9 = 22%, slightly higher than 15%, which is acceptable variance for a short sentence.

Step 4: Apply the 80-10-10 rule

Roll the dice for Word 4: suppose the outcome is 0.73 (less than 0.80), so we apply masking. Both "undeni" and "##ably" become [MASK]:

Original: ["The", "transform", "##ation", "was", "undeni",    "##ably",   "remark", "##able", "."]
Masked:   ["The", "transform", "##ation", "was", "[MASK]",    "[MASK]",   "remark", "##able", "."]
Labels:   [ -100,        -100,       -100,  -100, <id_undeni>, <id_ably>,    -100,     -100,  -100]

The label array stores the original token IDs at masked positions and -100 everywhere else. During training, the loss is computed only at positions with non-(-100) labels.

Step 5: Observe the learning signal

The model receives the masked sequence and must predict the tokens at positions 4 and 5. The visible context is: "The transformation was [MASK] [MASK] remarkable." To predict what fills the blank, the model must reason about what adverb-adjective combination could fit between "was" and "remarkable". This requires contextual interpretation: something that means "completely, without doubt, certainly" makes sense here. The model cannot rely on morphological patterns because neither the stem nor the suffix is visible. This is precisely the learning signal we want.