Masked Language Modeling

What if a model could see the future? Causal language modeling enforces a strict left-to-right constraint: each prediction depends only on preceding tokens. But natural language understanding often requires context from both directions. The word "bank" in "I deposited money at the bank" means something different than in "I sat by the river bank." Resolving such ambiguities requires seeing the full sentence.

Masked Language Modeling (MLM) removes the unidirectional constraint by hiding random tokens and asking the model to reconstruct them from surrounding context. This bidirectional approach, introduced with BERT in 2018, produces representations that capture meaning more effectively than left-to-right models for many understanding tasks. The trade-off is that MLM models cannot generate text autoregressively, making them specialists in comprehension rather than production.

The key insight behind MLM is deceptively simple: you can teach a model about language by playing a fill-in-the-blank game at massive scale. Given billions of sentences, a model that learns to fill in missing words must develop a sophisticated internal model of grammar, semantics, world knowledge, and pragmatics. Every successful prediction rewards the model for integrating contextual clues from the full sentence. Every failed prediction pushes the model to pay closer attention to linguistic patterns. No human annotation is required. The training signal comes directly from the structure of language itself.

Think of MLM as a form of self-supervised learning. The model creates its own supervision signal by corrupting the input and then trying to recover it. This means you can train on any raw text, in any language, on any domain, without paying annotators. The internet contains enough text to keep a model training for years, and none of it requires labeling. This self-supervised approach enabled the scale that made models like BERT and RoBERTa, plus their descendants, so powerful.

In this chapter, we'll explore the MLM objective, understand the masking strategies that make it work, implement the training procedure, and examine when bidirectional context matters most. We'll also build intuition for why these architectural choices produce better representations for understanding tasks and where the approach reaches its limits.

The core insight behind MLM is that understanding a word often requires seeing what comes after it, not just what came before. Consider the sentence:

The scientist studied the cell under a microscope.

When predicting "cell," a left-to-right model sees only "The scientist studied the." This provides some signal, but "cell" could still mean a prison cell, a biological cell, or a spreadsheet cell. The word "microscope" appearing later disambiguates completely, pointing to the biological meaning.

MLM allows the model to use this future context. By masking "cell" and asking the model to predict it, we force the model to integrate information from both "scientist studied" and "under a microscope" to make the prediction. The result is representations that encode richer semantic relationships.

The bidirectional advantage matters most for words whose meaning depends heavily on what follows them. In English, verbs often appear before their objects, so knowing the object changes how you interpret the verb. In German and many other languages, the verb comes at the end of the clause, making unidirectional left-to-right models especially unsuited for understanding. In all languages, pronouns and anaphora require looking backward and forward to resolve references. A sentence like "The trophy didn't fit in the suitcase because it was too big" requires understanding whether "it" refers to the trophy or the suitcase, a judgment that demands processing the entire sentence simultaneously.

Consider a few more examples where future context is decisive. In "I need to book a flight" versus "I need to read a book," the word "book" has different meanings that become clear from subsequent context. In "The bank failed" versus "The bank by the river flooded," the ambiguity resolves only when you see the rest of the sentence. In question answering, the model often needs to connect a question word to an answer that appears much later in the passage. All of these tasks benefit from full bidirectional context.

Notice that this advantage comes with a cost. Because the model sees the entire sequence at once, it cannot generate text one token at a time by conditioning on its own previous outputs. MLM is fundamentally a discriminative, not generative, pretraining objective. The model learns to score and fill in text, not to produce it from scratch. This design choice shapes everything downstream: what tasks the model excels at, how you fine-tune it, and what applications you can build.

This bidirectional context is the key advantage of MLM. For classification, entailment, question answering, and other understanding tasks, seeing the full context produces better representations than the partial view available to autoregressive models. When you fine-tune a BERT-style model on a classification task, every token in the sequence has already attended to every other token during the forward pass. By the time you read out the representation from the [CLS] token, it encodes information from the entire input rather than a running summary of what appeared so far.

How do we translate the intuition of "hide and predict" into a training objective? The answer involves three connected ideas: selecting which tokens to hide, defining what the model should predict, and measuring how well it succeeds. Let's build up the formalism step by step.

Consider a sentence like "The cat sat on the mat." We want to train a model that can recover hidden words from context. The training procedure works as follows:

1. Start with a complete sequence: We have $x = (x_1, x_2, \ldots, x_n)$, a sequence of $n$ tokens
2. Select positions to mask: We randomly choose a subset of positions $\mathcal{M} \subset \{1, \ldots, n\}$
3. Corrupt the input: We create $\tilde{x}$ by replacing tokens at masked positions with [MASK]
4. Predict the originals: The model must recover the original tokens at masked positions using the remaining context

The key insight is that step 4 requires the model to understand language deeply. To predict a masked word, the model must integrate syntactic constraints (what part of speech fits here?), semantic relationships (what meaning makes sense?), and world knowledge (what's plausible in this context?).

Think of it this way: if you give a human the sentence "The cat [MASK] on the mat," they can easily fill in "sat" or "lay" or "slept." They draw on knowledge that cats rest on flat surfaces, that the surrounding words are consistent with a resting verb, and that the sentence structure calls for a past-tense verb in that position. A model that learns to do this at scale must encode all of those same patterns in its weights. The masking task is a proxy objective that forces the model to build exactly the kind of rich, context-sensitive representations that transfer to downstream understanding tasks.

Worked Example. Take the sentence "The surgeon performed the [MASK] with great precision." A model processing this example sees:

• Left context: "The surgeon performed the," which points to a medical procedure
• Right context: "with great precision," which indicates a skilled, technical action

The model must integrate both sides. The left context alone would accept many nouns ("The surgeon performed the dance"). The right context alone is consistent with anything skilled ("The chef performed the filleting with great precision"). Together, they strongly suggest a word like "operation" or "surgery." To make that prediction correctly, the model must have learned that surgeons perform operations, that the phrase "with great precision" collocates with technical procedures, and that the article "the" implies a specific, countable event. The masking objective is deceptively simple to define but demands deep linguistic competence to solve.

We formalize "predict the originals" as maximizing the probability the model assigns to the correct tokens. Given the corrupted sequence $\tilde{x}$, for each masked position $i \in \mathcal{M}$, we want:

where $P_\theta(x_i | \tilde{x})$ is the probability the model with parameters $\theta$ assigns to the original token $x_i$, conditioned on seeing the entire corrupted sequence $\tilde{x}$. Note the conditioning: the model sees all of $\tilde{x}$, including tokens both before and after position $i$. This is the bidirectional context that distinguishes MLM from causal LM.

To combine predictions across all masked positions into a single training signal, we sum their log-probabilities:

where:

• $\mathcal{L}_{\text{MLM}}$: the masked language modeling loss we want to minimize
• $\mathcal{M}$: the set of masked position indices (typically $|\mathcal{M}| \approx 0.15n$, about 15% of positions)
• $x_i$: the original token at position $i$ that we want to recover
• $\tilde{x}$: the corrupted input sequence where tokens at positions in $\mathcal{M}$ have been replaced
• $P_\theta(x_i | \tilde{x})$: the probability the model assigns to the correct token, given bidirectional context
• $\log P_\theta(x_i | \tilde{x})$: the log-probability, which is negative since probabilities lie in $(0, 1)$

The formula uses logarithms for two reasons. First, products of probabilities become sums of log-probabilities, which are more numerically stable. When you compute the joint probability of $n$ correct predictions, you multiply $n$ small numbers together. For sequences of any meaningful length, this product underflows to zero in floating-point arithmetic. Taking logarithms maps the product into a sum, which remains numerically tractable. Second, the logarithm creates a useful asymmetry in the loss signal.

When the model is confident and correct ($P \approx 1$), we have $\log(1) = 0$, contributing zero loss. When the model is uncertain ($P \approx 0.5$), we have $\log(0.5) \approx -0.69$, contributing moderate loss. When the model is wrong ($P \approx 0.01$), we have $\log(0.01) \approx -4.6$, contributing large loss.

In practice, this asymmetry means the gradient is largest when the model is most wrong, which is exactly when it most needs to update. A model that confidently assigns 99% probability to the correct token gets almost no gradient signal, letting that token "coast" while the optimizer focuses on harder predictions. A model that assigns low probability to the correct token receives a strong gradient, pulling the parameters toward better representations of the surrounding context. This behavior makes negative log-likelihood a particularly well-suited training signal for prediction tasks.

The negative sign in front of the sum flips these negative log-probabilities into positive loss values. Minimizing this loss pushes the model toward assigning high probability to the correct tokens.

The summation in the MLM loss iterates only over masked positions $i \in \mathcal{M}$, not all positions. This is fundamentally different from causal LM, where every position contributes to the loss. In CLM, predicting position 5 uses only positions 1-4. In MLM, predicting position 5 uses positions 1-4 and positions 6 onward, but only if position 5 is masked.

This trade-off has practical consequences: MLM is less sample-efficient per token (only 15% of positions contribute gradients), but each prediction benefits from richer context. The bidirectional signal compensates for the sparsity, producing representations that excel at understanding tasks.

There is also a subtle difference in what the two objectives learn to represent. A CLM model builds representations that are, by construction, useful for predicting the next token. Every hidden state is shaped by the task of extending the sequence. An MLM model, by contrast, builds representations that are useful for understanding a token in context, without any specific pressure to predict what comes next. This difference in objective leads to a measurable difference in the representations' properties: MLM embeddings tend to cluster words by semantic role and syntactic function, while CLM embeddings are shaped more by sequential co-occurrence patterns. For downstream classification and understanding tasks, the MLM representations provide a better starting point for fine-tuning.

The original BERT paper established 15% as the masking rate: for each training example, approximately 15% of tokens are selected for prediction. This choice balances two competing concerns.

Masking too few tokens wastes compute. If only 1% of tokens are masked, 99% of the forward pass contributes nothing to the loss. The model processes the full sequence but learns from almost none of it. Even with the best representations, most of the work done by the transformer layers goes unrewarded. Training becomes extraordinarily expensive per useful gradient update.

Masking too many tokens destroys context. If 50% of tokens are masked, the model must predict half the sequence from the other half. With so much information missing, predictions become guesses rather than informed inferences. At extreme masking rates, the task changes character: instead of filling in a word from rich context, the model is essentially reconstructing a document from fragments, which requires different and arguably less useful capabilities.

The 15% rate emerged from empirical tuning in the original BERT paper. It provides enough masked tokens to learn efficiently while preserving enough context for accurate predictions. Later work has explored dynamic masking rates, but 15% remains the default for most MLM training. SpanBERT (2020) explored masking contiguous spans of tokens rather than random individual positions, finding that span masking forces the model to learn richer representations of phrases and multi-word expressions. The core trade-off, however, remains the same regardless of what kind of tokens are masked.

One important nuance: the 15% selection rate is applied to the already-tokenized sequence, not to words. For a subword tokenizer like WordPiece, a single long word like "uncharacteristically" might tokenize into five or six subword pieces. If one of those pieces is selected for masking, only that piece is masked, not the entire word. This means the model can often recover the masked piece trivially from the remaining pieces of the same word. Whole-word masking, which we'll cover in the next chapter, addresses exactly this issue by masking all subword tokens corresponding to the same word whenever that word is selected.

Simply replacing all selected tokens with [MASK] creates a mismatch between training and inference. During training, the model sees [MASK] tokens everywhere. During fine-tuning and inference, it never sees them. This discrepancy can hurt transfer performance because the model learns to treat [MASK] as a cue to do its best prediction work, and then loses that cue entirely after pretraining.

BERT addresses this with the 80-10-10 rule. For tokens selected for prediction:

• 80% are replaced with [MASK]
• 10% are replaced with a random token
• 10% are kept unchanged

Each of these three cases serves a distinct purpose, and understanding them reveals the careful reasoning behind the masking design.

The 80% [MASK] case is the primary training signal. Most selected tokens become [MASK], forcing the model to predict the original word from context. This is the core task that drives representation learning. The majority weight ensures that predictions of masked tokens dominate the gradient, keeping the model focused on the primary objective.

The 10% random replacement case trains the model to handle noisy input. When a random token is substituted, the model cannot simply trust that visible tokens are correct. It must learn to represent each token in terms of its identity and how well it fits its context. This is subtle but powerful: even tokens that are not masked must be understood in relation to their neighbors, because any of them might be noisy. As a result, the model's representations become less token-centric and more context-sensitive. Notice that the model still has to output the original token at this position, not the random one, so the loss correctly penalizes the model for believing the random intruder.

The 10% unchanged case closes a different gap. Without this case, the model would learn to only generate meaningful predictions for positions marked by [MASK] and to apply only superficial processing to unmasked positions. By requiring prediction even on unchanged tokens, the model cannot "save" its processing for masked positions. It must maintain rich representations everywhere. In practice, this means that during fine-tuning, when no masking occurs, every token still receives a high-quality contextual representation because the model was trained to produce them uniformly.

The output shows how masking transforms the input. Positions where labels equals -100 are not masked and won't contribute to the loss. Positions with non-negative labels are masked positions where the model must predict the original token. some masked positions show the [MASK] token ID (103), while others show random tokens or remain unchanged. This reflects the 80-10-10 strategy.

The 80-10-10 strategy forces the model to:

1. Learn to use context to recover masked tokens (the 80% case)
2. Learn representations that tolerate noisy input (the 10% random case)
3. Learn that unchanged tokens might still need prediction (the 10% unchanged case)

This last point is subtle but important. By sometimes requiring predictions on unchanged tokens, the model cannot simply "copy" visible tokens to the output. It must use context even for tokens that appear unmodified.

The combined effect of all three cases is that the model's encoder learns to produce high-quality, context-sensitive representations for every token in the sequence, not just for the masked ones. This is exactly what you want for a pretrained encoder that will be fine-tuned on classification, entailment, or retrieval tasks: representations that are rich and stable, as well as independent of any particular masking pattern.

MLM and CLM produce fundamentally different models suited for different tasks. The choice between them defines what the model can do, how it encodes language internally, and what kind of representations emerge from training.

Causal LM excels at generation because it models the natural process of producing text token by token. Each prediction extends the sequence, and the model can generate indefinitely by sampling from its predictions. The architecture enforces a causal constraint: position $i$ cannot attend to positions $j > i$. This means the hidden state at position $i$ encodes everything the model knows about the prefix $x_1, \ldots, x_i$, which is exactly the information you need to predict $x_{i+1}$. GPT and LLaMA, along with most chatbots, use CLM.

Masked LM excels at understanding because it captures relationships in both directions. For classification, the model can integrate information from the entire input before making a decision. For question answering, it can match question words with answer words regardless of their positions. For named entity recognition, it can determine a token's role from both its left and right neighbors. BERT and RoBERTa, along with most embedding models, use MLM.

The key reason MLM representations are better for understanding tasks is that each position's hidden state is conditioned on the full sequence. When you extract the [CLS] token's representation for classification, that vector has already attended to every other token in the input. When you extract a span's representation for question answering, those vectors encode bidirectional context. A CLM model's final hidden states are also conditioned on the entire prefix, but the model was never trained to produce representations that are useful for tasks requiring symmetric, full-sequence understanding.

Table mlm-vs-clm summarizes the key differences:

Neither approach is universally better. They're different tools optimized for different jobs. In practice, many systems combine both: an MLM encoder for understanding input, and a CLM decoder for generating output. T5 and BART take exactly this approach. T5 converts every NLP task into a text-to-text format and trains an encoder-decoder architecture where the encoder uses full bidirectional attention (like MLM) and the decoder uses causal attention (like CLM). BART pre-trains by corrupting documents with various noising functions and training the model to reconstruct the original, giving the encoder bidirectional context while training the decoder autoregressively. These hybrid architectures capture the representational advantages of both paradigms.

Let's implement a complete MLM training loop. We'll use a small transformer and train on sample text to see the dynamics in action. The implementation illuminates several key architectural decisions: the absence of a causal mask in the attention layers, the structure of the prediction head, and how the training loss ignores non-masked positions using a special ignore index.

MLM training differs from CLM in several important ways that affect how models learn. Understanding these dynamics helps you make better decisions about training hyperparameters, evaluate whether a model is learning effectively, and identify common failure modes.

Because only 15% of tokens are masked, only 15% of the output positions contribute to the gradient. This is less sample-efficient than CLM, where every position provides signal. To compensate, MLM models typically train for more steps or on more data.

The sparsity also affects optimization. In CLM, every forward pass provides dense feedback across the entire sequence, allowing the optimizer to update many parameters simultaneously. In MLM, most of the sequence is invisible to the loss. The model processes the full sequence for every forward pass, expending compute on unmasked tokens that contribute nothing to the gradient. This is the computational cost of bidirectionality: you must run the full bidirectional attention over the entire sequence even when only a small fraction of positions receive gradient updates.

CLM suffers from "exposure bias": during training, the model always sees ground truth previous tokens, but during generation, it sees its own predictions. This mismatch can cause errors to compound. A model trained with teacher forcing (always seeing the correct previous token) may not learn to recover from its own mistakes, leading to hallucinations and degradation over long generation sequences.

MLM doesn't have this problem because it doesn't generate autoregressively. The model always conditions on the full (corrupted) input, both during training and inference. The masking introduces a different kind of corruption, but it is applied identically at training time and at inference time for tasks like masked filling. MLM representations therefore tolerate corrupted context during understanding tasks: the model never develops a dependency on being given an entirely correct context.

The downside is that MLM is not directly applicable to generation. When researchers want to use a BERT-style model for translation or summarization, they must either add a decoder (as in BART) or convert the task into a filling problem. Neither approach is as natural as simply prompting a CLM model.

In MLM, predictions at different masked positions are made independently, in parallel. The model predicts all masked tokens simultaneously, not sequentially. This is efficient for training but means the model doesn't capture dependencies between masked tokens.

Consider masking both "New" and "York" in "I visited New York." CLM would predict "York" conditional on having already predicted "New." MLM predicts both independently, potentially outputting "New Orleans" and "Los Angeles" as individual predictions. The model might correctly understand that "New" suggests a city name and that "York" makes sense in that context, but it has no mechanism to ensure that the two predictions are consistent with each other.

This limitation is one of the motivations for XLNET (2019), which introduced permutation language modeling. Instead of masking tokens, permutation LM trains on randomly permuted orderings of the sequence, allowing the model to predict each token from any subset of the others. This captures both the bidirectional context of MLM and the autoregressive dependency structure of CLM, though at significantly higher computational cost.

The original BERT used static masking: each training example had the same tokens masked throughout training. The masking was applied once during data preprocessing and saved to disk. Every time the model saw a given sentence, the same tokens were masked. RoBERTa (2019) introduced dynamic masking, where masking is applied fresh for each training step, so the same sentence receives different masks across different passes through the dataset.

With static masking, the same tokens are masked every time the model sees this sequence. With dynamic masking, different tokens are masked on each pass, exposing the model to more varied training signal from the same data. Dynamic masking provides more variety during training. The model sees the same underlying sequences but with different tokens masked, effectively multiplying the diversity of training signal. RoBERTa showed this simple change improves downstream performance, especially with longer training.

The benefit of dynamic masking scales with the number of training epochs. If you train for 10 epochs with static masking, the model sees each mask pattern 10 times, gaining little new information from each repeat. With dynamic masking, each of those 10 passes presents a different subset of tokens as the prediction target, so every epoch contributes a new learning signal. The RoBERTa paper trained for far longer than BERT and found that the combination of dynamic masking, larger batch sizes, and more data consistently improved downstream task performance. Dynamic masking was one of several "obvious" improvements that turned out to matter more than the original BERT had suggested.

The primary use of MLM is learning representations that transfer to downstream tasks. After pretraining, the model's hidden states capture rich semantic information that can be fine-tuned for classification, question answering, named entity recognition, and other tasks. The pretrain-then-fine-tune paradigm that BERT established has become the dominant approach for NLP and has since been adopted in computer vision and other domains.

The typical workflow is:

1. Pretrain on large unlabeled corpus with MLM objective
2. Fine-tune on labeled data for specific task
3. Infer using the fine-tuned model

During fine-tuning, the [MASK] token is never used. The model processes normal text and uses its pretrained representations as a starting point. Fine-tuning updates all weights to adapt to the specific task. The MLM head is discarded and replaced with a task-specific head: a linear layer for classification, a span-extraction layer for question answering, or a token-level classifier for named entity recognition.

The reason fine-tuning works so well is that MLM pretraining has already solved the hardest part: learning what language means. By the time fine-tuning begins, the model's encoder encodes syntactic roles and semantic relationships, plus coreference and world knowledge, in its hidden states. Fine-tuning then needs only to learn a relatively simple mapping from these rich representations to task outputs. This is why fine-tuning on small labeled datasets (hundreds to thousands of examples) can still produce high-quality models: the heavy lifting has already been done.

Consider the difference between fine-tuning a BERT model on sentiment classification versus training a sentiment classifier from scratch. From scratch, you must learn simultaneously how to represent words and sentences, as well as discourse, and how those representations map to sentiment. With BERT, you only need to learn the last mapping. The task-specific training signal does not need to reconstruct the entire representation space, only to adjust it for the specific task. This is why BERT-style models dominated NLP benchmarks for several years after their introduction.

The representations produced by pretrained MLM models are also valuable for tasks beyond supervised classification. Sentence embeddings derived from MLM encoders power semantic search systems that find documents matching a query even when no exact keywords match. Similarity scores between embeddings enable document clustering, duplicate detection, and recommendation. The [CLS] token's representation, which aggregates information from the entire input, provides a natural sentence-level embedding for downstream use.

Masked language modeling has transformed NLP, but it comes with fundamental limitations that shape its applications. Understanding these constraints is essential for making good architectural choices when building systems.

The inability to generate text is the most significant constraint. MLM models cannot produce coherent sequences token by token. They can fill in blanks and score existing text, but they cannot write. This limitation means MLM is unsuitable for chatbots, story generation, code completion, and other generative applications. The underlying reason is architectural: generation requires sequential conditioning, where each new token depends on previously generated tokens. MLM models are trained to condition on the full sequence simultaneously, so there is no natural way to generate left to right. The distinction between understanding and generation has driven the field toward hybrid architectures that combine MLM-style encoders with CLM-style decoders, as seen in T5 and BART.

The masking mismatch between pretraining and fine-tuning creates subtle issues. During pretraining, 15% of tokens are corrupted, and the model sees [MASK] tokens frequently. During fine-tuning, no tokens are masked, and the model never sees [MASK] again. The 80-10-10 strategy mitigates this by keeping 10% of selected tokens unchanged and 10% random, but the model still encounters [MASK] during 80% of masked positions in pretraining and then never during fine-tuning. Research on continuous masking and better pretraining objectives continues to address this gap. ELECTRA (2020) took a fundamentally different approach: instead of masking tokens, it trains a small generator to replace tokens with plausible-sounding alternatives, then trains a discriminator to detect which tokens were replaced. The discriminator receives a signal at every token position (not just masked ones), addressing both the sample efficiency and the masking mismatch simultaneously.

Sample efficiency is another concern. With only 15% of tokens contributing to the loss, MLM requires more compute than CLM to see the same amount of training signal. RoBERTa compensated by training longer and on more data, but this increases cost. ELECTRA showed that the discriminative pretraining signal (every token is real or replaced) converges to better representations with the same compute budget as BERT. Recent work on efficient pretraining explores higher masking rates, span masking, and alternative objectives, but the fundamental tension between bidirectionality and gradient density has not been resolved.

Scaling behavior is also less favorable than CLM. CLM models benefit enormously from scale: larger models with more data consistently produce better generation quality, and many of the most impressive capabilities of GPT-4 and similar models emerge only at large scale. MLM models also improve with scale, but the gains on understanding benchmarks plateau earlier, and the most useful MLM models tend to be in the hundreds-of-millions range rather than the hundreds-of-billions range. The reasons for this difference are still being studied, but it likely relates to the sparse gradient signal and the discriminative nature of MLM pretraining.

Despite these limitations, MLM enabled capabilities that were previously difficult to achieve. BERT's bidirectional representations set new state-of-the-art results on eleven NLP benchmarks when released, including GLUE and SQuAD, as well as CoNLL NER. The pretrain-then-fine-tune paradigm it established remains the dominant approach for understanding tasks. Sentence embeddings from MLM models power semantic search, document clustering, and similarity computations across the industry. Models like Sentence-BERT and instructor-xl, both built on MLM foundations, enable retrieval systems that understand meaning rather than just matching keywords. The insight that bidirectional context improves understanding has influenced the design of virtually every encoder since, and the MLM objective continues to be used in domain-specific pretraining for scientific and medical, plus legal, text.

When training MLM models, several parameters significantly impact performance. Understanding what each parameter controls and why the canonical values were chosen helps you make principled decisions when adapting MLM training to new domains or resource constraints.

The core training parameters are:

• mask_prob (default: 0.15): Fraction of tokens to mask per sequence. Higher values provide more training signal but destroy more context. The 15% rate from BERT remains standard, though some work explores 40% or higher with adjusted strategies. SpanBERT found that masking contiguous spans with a geometric length distribution performs better than uniform token masking, even at the same 15% rate.
• d_model: Hidden dimension of the transformer. BERT-base uses 768, BERT-large uses 1024. Larger values increase model capacity but require proportionally more compute and memory. The hidden dimension controls how much information each token's representation can encode. Too small, and the model cannot represent complex contextual relationships. Too large, and the model becomes expensive to deploy.
• n_heads: Number of attention heads. Should divide d_model evenly. BERT-base uses 12 heads (64 dimensions each), BERT-large uses 16 heads. Each attention head can specialize in a different type of relationship: some heads learn syntactic dependencies, others learn semantic associations, and still others track coreference chains. Having more heads gives the model more capacity to represent diverse linguistic relationships in parallel.
• n_layers: Number of transformer layers. BERT-base uses 12, BERT-large uses 24. Deeper models capture more complex patterns but are slower to train and infer. Probing studies have shown that lower layers tend to encode syntactic information, while higher layers encode more abstract semantic content. This stratification is one reason why removing the top few layers (or using intermediate layer representations) sometimes improves performance on certain tasks.
• max_len: Maximum sequence length the model can process. BERT uses 512 tokens. Longer contexts require quadratically more memory for attention but capture more context. Most real-world NLP tasks fit within 512 tokens, but document-level tasks like long-form question answering and summarization benefit from longer contexts.
• learning_rate: Typically 1e-4 to 5e-4 for MLM pretraining. BERT used 1e-4 with a linear warmup over the first 1% of training steps. Higher rates speed training but risk instability, especially for large models. RoBERTa found that larger batch sizes combined with higher learning rates (up to 4e-4) accelerated convergence without sacrificing quality.
• batch_size: Larger batches provide more stable gradients. BERT used effective batch sizes of 256 sequences. MLM benefits from large batches since only 15% of tokens contribute to each gradient. RoBERTa trained with batch sizes of 8,000 sequences, finding that large batch training at high learning rates significantly reduced training time while achieving better final performance.

Masked language modeling trains models to predict randomly masked tokens from bidirectional context. The objective is simple to state but demands sophisticated language understanding to solve at scale. This chapter covered the key concepts:

• Bidirectional context allows MLM models to use information from both before and after each position, producing richer representations than unidirectional models. This bidirectionality is the core advantage for understanding tasks.
• The 15% masking rate balances sample efficiency against context preservation. This provides enough training signal while keeping most context visible. Too little masking wastes compute; too much destroys the context needed for accurate prediction.
• The 80-10-10 strategy (80% [MASK], 10% random, 10% unchanged) reduces the mismatch between pretraining and fine-tuning by exposing the model to real tokens at training time, forcing it to produce good representations everywhere, not just at masked positions.
• MLM vs. CLM represents a fundamental trade-off: MLM excels at understanding tasks while CLM excels at generation. Hybrid architectures like T5 and BART combine both to handle both understanding and generation.
• Dynamic masking applies fresh masks each epoch, increasing training signal diversity. The RoBERTa results showed this simple change produces meaningful improvements, especially with longer training runs.
• Sparse gradients from masking only 15% of positions make MLM less sample-efficient than CLM. This is the price of bidirectionality.
• Independent predictions at masked positions mean MLM cannot model dependencies between masked tokens. XLNET's permutation language modeling addresses this at higher computational cost.
• The pretrain-then-fine-tune paradigm that BERT established remains the dominant approach for NLP understanding tasks, enabling high-quality task-specific models from small labeled datasets.

The next chapter explores whole word masking, a refinement that improves MLM by masking entire words rather than individual subword tokens, addressing a key weakness in how random subword masking interacts with WordPiece tokenization.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about masked language modeling.