RoBERTa › Dynamic Masking
Language AI HandbookAll chapters RoBERTa › Dynamic MaskingOne section · Sections in this chapter4 / 17 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
BERT used static masking: the training data was preprocessed once, with masks applied and saved. Each training example always had the same tokens masked, even across multiple epochs. This limited the diversity of training signal.
RoBERTa introduced dynamic masking: masks are generated on-the-fly during training. Each time the model sees a sequence, different tokens are masked. Over multiple epochs, the model sees the same underlying text with many different masking patterns.
The difference might seem like a minor engineering detail, but it has real consequences for what the model learns. Think of it this way: if a student studies the same highlighted passages in a textbook every time they review it, they will become very good at recalling those particular phrases. But if the highlights change each time, forcing the student to predict different parts from different context, they develop a more complete understanding of the entire passage. Dynamic masking pushes the model toward this second, more thorough mode of learning.
In BERT's static masking approach, the 15% of tokens that are masked remain the same every epoch. A token that was never masked during preprocessing will never be predicted by the model during pretraining, regardless of how many epochs training runs. For any given sequence, some tokens are perpetually "safe" and always appear as context, while others are always "targets." This creates an asymmetry: the model becomes expert at predicting the specific small set of masked tokens, but the remaining 85% of tokens exist only as context input, never as prediction targets during that training run.
Dynamic masking dissolves this asymmetry. With fresh random masks each epoch, every token in the corpus eventually becomes a prediction target, given enough training. The model must learn to predict any token from any surrounding context, including tokens that were not selected during preprocessing. This broader coverage contributes to better representations.
import torch
def static_masking_preprocess(token_ids, mask_token_id=103, mask_prob=0.15):
"""
BERT-style static masking: Apply once during data preprocessing.
The same positions are masked every time this example is used.
"""
masked_ids = token_ids.clone()
labels = token_ids.clone()
# Determine which positions to mask (this is fixed)
probability_matrix = torch.full(token_ids.shape, mask_prob)
masked_indices = torch.bernoulli(probability_matrix).bool()
# Non-masked positions get label -100 (ignored in loss)
labels[~masked_indices] = -100
# Apply 80-10-10 strategy
indices_replaced = (
torch.bernoulli(torch.full(token_ids.shape, 0.8)).bool()
& masked_indices
)
masked_ids[indices_replaced] = mask_token_id
indices_random = (
torch.bernoulli(torch.full(token_ids.shape, 0.5)).bool()
& masked_indices
& ~indices_replaced
)
random_tokens = torch.randint(1000, token_ids.shape, dtype=token_ids.dtype)
masked_ids[indices_random] = random_tokens[indices_random]
return masked_ids, labels
def dynamic_masking(token_ids, mask_token_id=103, mask_prob=0.15):
"""
RoBERTa-style dynamic masking: Apply fresh masks each training step.
Different positions are masked each time, increasing training diversity.
"""
# Exactly the same logic, but called each time during training
# rather than once during preprocessing
return static_masking_preprocess(token_ids, mask_token_id, mask_prob)Let's visualize the difference over multiple training epochs:
# Simulate 4 epochs of seeing the same sequence
torch.manual_seed(42)
original_sequence = torch.tensor([101, 2023, 2003, 1037, 3231, 6251, 2517, 102])
# Static masking: preprocess once
static_masked, static_labels = static_masking_preprocess(
original_sequence.clone()
)
# Dynamic masking: generate fresh masks each epoch
dynamic_epochs = []
for epoch in range(4):
torch.manual_seed(epoch * 100 + 42) # Different seed each epoch
masked, labels = dynamic_masking(original_sequence.clone())
dynamic_epochs.append((masked.tolist(), labels.tolist()))Original sequence: [101, 2023, 2003, 1037, 3231, 6251, 2517, 102] --- Static Masking (BERT) --- Epoch 1: [101, 2023, 2003, 1037, 3231, 6251, 2517, 102] Epoch 2: [101, 2023, 2003, 1037, 3231, 6251, 2517, 102] Epoch 3: [101, 2023, 2003, 1037, 3231, 6251, 2517, 102] Epoch 4: [101, 2023, 2003, 1037, 3231, 6251, 2517, 102] --- Dynamic Masking (RoBERTa) --- Epoch 1: [101, 2023, 2003, 1037, 3231, 6251, 2517, 102] (masked positions: []) Epoch 2: [101, 2023, 2003, 1037, 3231, 6251, 103, 102] (masked positions: [6]) Epoch 3: [101, 2023, 103, 1037, 3231, 6251, 2517, 102] (masked positions: [2]) Epoch 4: [101, 2023, 2003, 1037, 3231, 6251, 2517, 102] (masked positions: [])
With static masking, the model sees identical inputs across all epochs. With dynamic masking, each epoch presents new challenges. Position 3 might be masked in epoch 1, positions 2 and 5 in epoch 2, and so on. This multiplies the effective training diversity without requiring more data.
Notice also that dynamic masking has a favorable interaction with the 80-10-10 substitution rule inherited from BERT. Recall that for each masked position, BERT and RoBERTa replace 80% with the actual [MASK] token, 10% with a random token, and leave 10% unchanged. With static masking, the 10% unchanged positions are fixed across all epochs: the same tokens are always left unchanged, always with their original identities exposed. With dynamic masking, the 10% unchanged positions change each epoch, meaning the model encounters different "leaking" tokens on each pass. This further diversifies the learning signal and reduces any systematic biases that might arise from fixed unchanged positions.
The empirical benefit of dynamic masking is modest but consistent. The RoBERTa paper found it slightly improves or matches static masking across all benchmarks. More importantly, it's simpler: there's no need to preprocess and store multiple masked versions of the data. Masking happens during training, reducing storage requirements and preprocessing complexity.
In practice, dynamic masking is the standard choice for any new pretraining project. Storage savings alone justify it: BERT's original implementation duplicated the training data multiple times to create different masking patterns for different epochs, consuming significant disk space. With dynamic masking, you store the raw tokenized text once and generate masks at training time. The compute overhead is negligible, the storage savings are substantial, and the quality is equal to or slightly better than the static alternative. This is a rare case where the simpler approach is also the better one.