Whole Word Masking › Whole Word Masking
Language AI HandbookAll chapters Whole Word Masking › Whole Word MaskingOne section · Sections in this chapter1 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
When BERT tokenizes "undeniably" into ["un", "##deni", "##ably"] and then masks only "##ably", something goes wrong. The model sees "un" and "##deni" in the clear, giving it strong hints about the masked portion. The prediction task becomes almost trivial: what word starts with "undeni-" and ends with a common suffix? This partial visibility undermines the entire learning objective and turns a sophisticated language modeling task into something closer to a morphology lookup table.
Whole Word Masking (WWM) fixes this by treating subword tokens as parts of atomic units. If any subword of a word is selected for masking, all subwords of that word are masked together. The model must now predict the entire word from surrounding context alone, without peeking at sibling subwords. This change improves downstream task performance, particularly on reading comprehension benchmarks and tasks that require semantic interpretation rather than pattern-matching at the subword level.
The idea behind WWM reflects a deeper principle: the unit of prediction should match the unit of meaning. Words carry meaning; subword fragments typically do not. When you mask "##ation" while leaving "transform" visible, you are asking the model to guess a suffix, not a meaning. When you mask "transformation" entirely, you are asking the model to reason about what concept fits the surrounding sentence. These are fundamentally different cognitive tasks, and only the second one trains the kind of contextual understanding that makes BERT useful downstream.
In this chapter, we examine why subword masking creates problems, how whole word masking works algorithmically at a precise level, and how to implement it correctly for different tokenizer families. We also look at the 80-10-10 masking rule and how it extends naturally to word boundaries, the challenge of maintaining a consistent masking ratio when word lengths vary, and the practical question of what happens in morphologically rich languages where word boundaries carry even more semantic weight. We then compare WWM against random subword masking with real BERT predictions to see the empirical differences in prediction difficulty.
Whole Word Masking was introduced by Google AI as a modification to the original BERT pre-training procedure in 2019, concurrent with the release of Chinese BERT models. The Chinese BERT paper (Cui et al., 2019) made the case most forcefully: because Chinese text has no spaces between characters, character-level masking in the original BERT was essentially random subword masking taken to an extreme, with individual characters masked independently from the words they composed. This severely limited the model's ability to learn word-level semantics. The whole word masking fix, which required an external word segmenter for Chinese, produced dramatic improvements on Chinese NLP benchmarks. Google subsequently released English BERT-wwm models that showed consistent though smaller improvements. The technique became standard for training non-English BERT variants, and its core insight, that masking units should align with semantic units, directly influenced later masking strategies including span masking and the corruption objectives used in T5 and other sequence-to-sequence pre-training frameworks.