Masked Language Modeling › The MLM Objective

How do we translate the intuition of "hide and predict" into a training objective? The answer involves three connected ideas: selecting which tokens to hide, defining what the model should predict, and measuring how well it succeeds. Let's build up the formalism step by step.

Consider a sentence like "The cat sat on the mat." We want to train a model that can recover hidden words from context. The training procedure works as follows:

1. Start with a complete sequence: We have $x = (x_1, x_2, \ldots, x_n)$, a sequence of $n$ tokens
2. Select positions to mask: We randomly choose a subset of positions $\mathcal{M} \subset \{1, \ldots, n\}$
3. Corrupt the input: We create $\tilde{x}$ by replacing tokens at masked positions with [MASK]
4. Predict the originals: The model must recover the original tokens at masked positions using the remaining context

The key insight is that step 4 requires the model to understand language deeply. To predict a masked word, the model must integrate syntactic constraints (what part of speech fits here?), semantic relationships (what meaning makes sense?), and world knowledge (what's plausible in this context?).

Think of it this way: if you give a human the sentence "The cat [MASK] on the mat," they can easily fill in "sat" or "lay" or "slept." They draw on knowledge that cats rest on flat surfaces, that the surrounding words are consistent with a resting verb, and that the sentence structure calls for a past-tense verb in that position. A model that learns to do this at scale must encode all of those same patterns in its weights. The masking task is a proxy objective that forces the model to build exactly the kind of rich, context-sensitive representations that transfer to downstream understanding tasks.

Worked Example. Take the sentence "The surgeon performed the [MASK] with great precision." A model processing this example sees:

• Left context: "The surgeon performed the," which points to a medical procedure
• Right context: "with great precision," which indicates a skilled, technical action

The model must integrate both sides. The left context alone would accept many nouns ("The surgeon performed the dance"). The right context alone is consistent with anything skilled ("The chef performed the filleting with great precision"). Together, they strongly suggest a word like "operation" or "surgery." To make that prediction correctly, the model must have learned that surgeons perform operations, that the phrase "with great precision" collocates with technical procedures, and that the article "the" implies a specific, countable event. The masking objective is deceptively simple to define but demands deep linguistic competence to solve.

We formalize "predict the originals" as maximizing the probability the model assigns to the correct tokens. Given the corrupted sequence $\tilde{x}$, for each masked position $i \in \mathcal{M}$, we want:

where $P_\theta(x_i | \tilde{x})$ is the probability the model with parameters $\theta$ assigns to the original token $x_i$, conditioned on seeing the entire corrupted sequence $\tilde{x}$. Note the conditioning: the model sees all of $\tilde{x}$, including tokens both before and after position $i$. This is the bidirectional context that distinguishes MLM from causal LM.

To combine predictions across all masked positions into a single training signal, we sum their log-probabilities:

where:

• $\mathcal{L}_{\text{MLM}}$: the masked language modeling loss we want to minimize
• $\mathcal{M}$: the set of masked position indices (typically $|\mathcal{M}| \approx 0.15n$, about 15% of positions)
• $x_i$: the original token at position $i$ that we want to recover
• $\tilde{x}$: the corrupted input sequence where tokens at positions in $\mathcal{M}$ have been replaced
• $P_\theta(x_i | \tilde{x})$: the probability the model assigns to the correct token, given bidirectional context
• $\log P_\theta(x_i | \tilde{x})$: the log-probability, which is negative since probabilities lie in $(0, 1)$

The formula uses logarithms for two reasons. First, products of probabilities become sums of log-probabilities, which are more numerically stable. When you compute the joint probability of $n$ correct predictions, you multiply $n$ small numbers together. For sequences of any meaningful length, this product underflows to zero in floating-point arithmetic. Taking logarithms maps the product into a sum, which remains numerically tractable. Second, the logarithm creates a useful asymmetry in the loss signal.

When the model is confident and correct ($P \approx 1$), we have $\log(1) = 0$, contributing zero loss. When the model is uncertain ($P \approx 0.5$), we have $\log(0.5) \approx -0.69$, contributing moderate loss. When the model is wrong ($P \approx 0.01$), we have $\log(0.01) \approx -4.6$, contributing large loss.

In practice, this asymmetry means the gradient is largest when the model is most wrong, which is exactly when it most needs to update. A model that confidently assigns 99% probability to the correct token gets almost no gradient signal, letting that token "coast" while the optimizer focuses on harder predictions. A model that assigns low probability to the correct token receives a strong gradient, pulling the parameters toward better representations of the surrounding context. This behavior makes negative log-likelihood a particularly well-suited training signal for prediction tasks.

The negative sign in front of the sum flips these negative log-probabilities into positive loss values. Minimizing this loss pushes the model toward assigning high probability to the correct tokens.

The summation in the MLM loss iterates only over masked positions $i \in \mathcal{M}$, not all positions. This is fundamentally different from causal LM, where every position contributes to the loss. In CLM, predicting position 5 uses only positions 1-4. In MLM, predicting position 5 uses positions 1-4 and positions 6 onward, but only if position 5 is masked.

This trade-off has practical consequences: MLM is less sample-efficient per token (only 15% of positions contribute gradients), but each prediction benefits from richer context. The bidirectional signal compensates for the sparsity, producing representations that excel at understanding tasks.

There is also a subtle difference in what the two objectives learn to represent. A CLM model builds representations that are, by construction, useful for predicting the next token. Every hidden state is shaped by the task of extending the sequence. An MLM model, by contrast, builds representations that are useful for understanding a token in context, without any specific pressure to predict what comes next. This difference in objective leads to a measurable difference in the representations' properties: MLM embeddings tend to cluster words by semantic role and syntactic function, while CLM embeddings are shaped more by sequential co-occurrence patterns. For downstream classification and understanding tasks, the MLM representations provide a better starting point for fine-tuning.

The original BERT paper established 15% as the masking rate: for each training example, approximately 15% of tokens are selected for prediction. This choice balances two competing concerns.

Masking too few tokens wastes compute. If only 1% of tokens are masked, 99% of the forward pass contributes nothing to the loss. The model processes the full sequence but learns from almost none of it. Even with the best representations, most of the work done by the transformer layers goes unrewarded. Training becomes extraordinarily expensive per useful gradient update.

Masking too many tokens destroys context. If 50% of tokens are masked, the model must predict half the sequence from the other half. With so much information missing, predictions become guesses rather than informed inferences. At extreme masking rates, the task changes character: instead of filling in a word from rich context, the model is essentially reconstructing a document from fragments, which requires different and arguably less useful capabilities.

The 15% rate emerged from empirical tuning in the original BERT paper. It provides enough masked tokens to learn efficiently while preserving enough context for accurate predictions. Later work has explored dynamic masking rates, but 15% remains the default for most MLM training. SpanBERT (2020) explored masking contiguous spans of tokens rather than random individual positions, finding that span masking forces the model to learn richer representations of phrases and multi-word expressions. The core trade-off, however, remains the same regardless of what kind of tokens are masked.

One important nuance: the 15% selection rate is applied to the already-tokenized sequence, not to words. For a subword tokenizer like WordPiece, a single long word like "uncharacteristically" might tokenize into five or six subword pieces. If one of those pieces is selected for masking, only that piece is masked, not the entire word. This means the model can often recover the masked piece trivially from the remaining pieces of the same word. Whole-word masking, which we'll cover in the next chapter, addresses exactly this issue by masking all subword tokens corresponding to the same word whenever that word is selected.