Masked Language Modeling › Summary

Masked language modeling trains models to predict randomly masked tokens from bidirectional context. The objective is simple to state but demands sophisticated language understanding to solve at scale. This chapter covered the key concepts:

• Bidirectional context allows MLM models to use information from both before and after each position, producing richer representations than unidirectional models. This bidirectionality is the core advantage for understanding tasks.
• The 15% masking rate balances sample efficiency against context preservation. This provides enough training signal while keeping most context visible. Too little masking wastes compute; too much destroys the context needed for accurate prediction.
• The 80-10-10 strategy (80% [MASK], 10% random, 10% unchanged) reduces the mismatch between pretraining and fine-tuning by exposing the model to real tokens at training time, forcing it to produce good representations everywhere, not just at masked positions.
• MLM vs. CLM represents a fundamental trade-off: MLM excels at understanding tasks while CLM excels at generation. Hybrid architectures like T5 and BART combine both to handle both understanding and generation.
• Dynamic masking applies fresh masks each epoch, increasing training signal diversity. The RoBERTa results showed this simple change produces meaningful improvements, especially with longer training runs.
• Sparse gradients from masking only 15% of positions make MLM less sample-efficient than CLM. This is the price of bidirectionality.
• Independent predictions at masked positions mean MLM cannot model dependencies between masked tokens. XLNET's permutation language modeling addresses this at higher computational cost.
• The pretrain-then-fine-tune paradigm that BERT established remains the dominant approach for NLP understanding tasks, enabling high-quality task-specific models from small labeled datasets.

The next chapter explores whole word masking, a refinement that improves MLM by masking entire words rather than individual subword tokens, addressing a key weakness in how random subword masking interacts with WordPiece tokenization.