Masked Language Modeling › Dynamic Masking
Language AI HandbookAll chapters Masked Language Modeling › Dynamic MaskingOne section · Sections in this chapter8 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The original BERT used static masking: each training example had the same tokens masked throughout training. The masking was applied once during data preprocessing and saved to disk. Every time the model saw a given sentence, the same tokens were masked. RoBERTa (2019) introduced dynamic masking, where masking is applied fresh for each training step, so the same sentence receives different masks across different passes through the dataset.
def static_masking(data, mask_prob=0.15):
"""Apply masking once, reuse throughout training."""
masked_ids, labels = apply_mlm_masking(
data.unsqueeze(0), vocab_size, mask_token_id, mask_prob
)
return masked_ids.squeeze(0), labels.squeeze(0)
def dynamic_masking(data, mask_prob=0.15):
"""Apply fresh masking each time."""
# This is called each time we need a batch
# Different tokens are masked each call
return static_masking(data, mask_prob)# Compare static vs dynamic masking on the same sequence
sample = encoded[:20]
static_masked, _ = static_masking(sample)Static masking (same pattern reused): [6, 14, 11, 2, 23, 27, 15, 9, 17, 2, 8, 24, 21, 29, 20, 2, 12, 0, 30, 0] [6, 14, 11, 2, 23, 27, 15, 9, 17, 2, 8, 24, 21, 29, 20, 2, 12, 0, 30, 0] [6, 14, 11, 2, 23, 27, 15, 9, 17, 2, 8, 24, 21, 29, 20, 2, 12, 0, 30, 0] Dynamic masking (fresh pattern each time): [6, 14, 11, 2, 23, 0, 15, 9, 17, 2, 8, 24, 21, 29, 0, 2, 12, 21, 30, 2] [6, 0, 11, 2, 23, 27, 15, 9, 17, 2, 8, 0, 21, 29, 20, 2, 12, 21, 30, 2] [6, 14, 11, 2, 0, 27, 15, 9, 17, 2, 8, 24, 21, 29, 0, 2, 12, 21, 30, 2]
With static masking, the same tokens are masked every time the model sees this sequence. With dynamic masking, different tokens are masked on each pass, exposing the model to more varied training signal from the same data. Dynamic masking provides more variety during training. The model sees the same underlying sequences but with different tokens masked, effectively multiplying the diversity of training signal. RoBERTa showed this simple change improves downstream performance, especially with longer training.
The benefit of dynamic masking scales with the number of training epochs. If you train for 10 epochs with static masking, the model sees each mask pattern 10 times, gaining little new information from each repeat. With dynamic masking, each of those 10 passes presents a different subset of tokens as the prediction target, so every epoch contributes a new learning signal. The RoBERTa paper trained for far longer than BERT and found that the combination of dynamic masking, larger batch sizes, and more data consistently improved downstream task performance. Dynamic masking was one of several "obvious" improvements that turned out to matter more than the original BERT had suggested.