Masked Language Modeling › Masked Language Modeling
Language AI HandbookAll chapters Masked Language Modeling › Masked Language ModelingOne section · Sections in this chapter1 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
What if a model could see the future? Causal language modeling enforces a strict left-to-right constraint: each prediction depends only on preceding tokens. But natural language understanding often requires context from both directions. The word "bank" in "I deposited money at the bank" means something different than in "I sat by the river bank." Resolving such ambiguities requires seeing the full sentence.
Masked Language Modeling (MLM) removes the unidirectional constraint by hiding random tokens and asking the model to reconstruct them from surrounding context. This bidirectional approach, introduced with BERT in 2018, produces representations that capture meaning more effectively than left-to-right models for many understanding tasks. The trade-off is that MLM models cannot generate text autoregressively, making them specialists in comprehension rather than production.
The key insight behind MLM is deceptively simple: you can teach a model about language by playing a fill-in-the-blank game at massive scale. Given billions of sentences, a model that learns to fill in missing words must develop a sophisticated internal model of grammar, semantics, world knowledge, and pragmatics. Every successful prediction rewards the model for integrating contextual clues from the full sentence. Every failed prediction pushes the model to pay closer attention to linguistic patterns. No human annotation is required. The training signal comes directly from the structure of language itself.
Before BERT (2018), the dominant pretraining approach was unidirectional. ELMo (2018) came close to bidirectionality by training two separate LSTMs (one left-to-right, one right-to-left) and concatenating their outputs, but each direction was trained independently. OpenAI GPT (2018) used a pure left-to-right transformer. BERT was the first model to train a single deep transformer with full bidirectional attention using the MLM objective, achieving state-of-the-art results on eleven NLP benchmarks simultaneously. The paper's title, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding," captured exactly what made it different: deep and bidirectional, not shallow or unidirectional.
Think of MLM as a form of self-supervised learning. The model creates its own supervision signal by corrupting the input and then trying to recover it. This means you can train on any raw text, in any language, on any domain, without paying annotators. The internet contains enough text to keep a model training for years, and none of it requires labeling. This self-supervised approach enabled the scale that made models like BERT and RoBERTa, plus their descendants, so powerful.
In this chapter, we'll explore the MLM objective, understand the masking strategies that make it work, implement the training procedure, and examine when bidirectional context matters most. We'll also build intuition for why these architectural choices produce better representations for understanding tasks and where the approach reaches its limits.