Masked Language Modeling › MLM Training Dynamics

MLM training differs from CLM in several important ways that affect how models learn. Understanding these dynamics helps you make better decisions about training hyperparameters, evaluate whether a model is learning effectively, and identify common failure modes.

Because only 15% of tokens are masked, only 15% of the output positions contribute to the gradient. This is less sample-efficient than CLM, where every position provides signal. To compensate, MLM models typically train for more steps or on more data.

The sparsity also affects optimization. In CLM, every forward pass provides dense feedback across the entire sequence, allowing the optimizer to update many parameters simultaneously. In MLM, most of the sequence is invisible to the loss. The model processes the full sequence for every forward pass, expending compute on unmasked tokens that contribute nothing to the gradient. This is the computational cost of bidirectionality: you must run the full bidirectional attention over the entire sequence even when only a small fraction of positions receive gradient updates.

CLM suffers from "exposure bias": during training, the model always sees ground truth previous tokens, but during generation, it sees its own predictions. This mismatch can cause errors to compound. A model trained with teacher forcing (always seeing the correct previous token) may not learn to recover from its own mistakes, leading to hallucinations and degradation over long generation sequences.

MLM doesn't have this problem because it doesn't generate autoregressively. The model always conditions on the full (corrupted) input, both during training and inference. The masking introduces a different kind of corruption, but it is applied identically at training time and at inference time for tasks like masked filling. MLM representations therefore tolerate corrupted context during understanding tasks: the model never develops a dependency on being given an entirely correct context.

The downside is that MLM is not directly applicable to generation. When researchers want to use a BERT-style model for translation or summarization, they must either add a decoder (as in BART) or convert the task into a filling problem. Neither approach is as natural as simply prompting a CLM model.

In MLM, predictions at different masked positions are made independently, in parallel. The model predicts all masked tokens simultaneously, not sequentially. This is efficient for training but means the model doesn't capture dependencies between masked tokens.

Consider masking both "New" and "York" in "I visited New York." CLM would predict "York" conditional on having already predicted "New." MLM predicts both independently, potentially outputting "New Orleans" and "Los Angeles" as individual predictions. The model might correctly understand that "New" suggests a city name and that "York" makes sense in that context, but it has no mechanism to ensure that the two predictions are consistent with each other.

This limitation is one of the motivations for XLNET (2019), which introduced permutation language modeling. Instead of masking tokens, permutation LM trains on randomly permuted orderings of the sequence, allowing the model to predict each token from any subset of the others. This captures both the bidirectional context of MLM and the autoregressive dependency structure of CLM, though at significantly higher computational cost.