Whole Word Masking › Summary

Whole Word Masking addresses a fundamental flaw in applying masked language modeling to subword-tokenized text. When subword tokens are masked independently, visible sibling tokens leak information about masked positions, reducing the prediction task to morphological pattern matching rather than contextual reasoning. This information leakage is not a small perturbation: it can reduce prediction entropy by a factor of five or more, dramatically weakening the gradient signal that drives learning.

WWM's core contributions are straightforward in principle and impactful in practice:

• Preserves word boundaries: All subwords of a word are masked together, eliminating within-word information leakage. The model must predict complete words from external context alone.

• Strengthens learning signal: Because the model cannot rely on visible sibling subwords, it must reason about semantic roles, syntactic structure, and pragmatic context to predict masked words. This produces richer gradient updates and stronger contextual representations.

• Adapts to tokenizer conventions: Different tokenizers mark word boundaries differently (## for WordPiece, ▁ for SentencePiece, Ġ for GPT-2 BPE), and WWM implementations should detect and handle each convention automatically.

• Integrates cleanly with existing frameworks: Libraries like Hugging Face provide ready-to-use WWM data collators that handle all implementation details, making it straightforward to add WWM to any existing MLM training pipeline.

• Has proven empirical value: Consistent improvements on reading comprehension and other benchmarks, particularly for morphologically rich languages, have made WWM the standard approach for non-English BERT pre-training.

The technique is particularly important for languages with rich morphology (German, Turkish, Finnish, Arabic) and for any domain where word-level understanding matters more than subword pattern matching (scientific text, legal documents, technical documentation). While the improvements over standard MLM are modest for English, they compound with other techniques and become essential for many non-English languages.

The next chapter explores span corruption, which extends the WWM insight further by masking contiguous spans of multiple words together, creating even more challenging prediction tasks that encourage models to learn longer-range dependencies and develop representations suited to sequence-to-sequence tasks.