Masked Language Modeling › The 80-10-10 Masking Strategy
Language AI HandbookAll chapters Masked Language Modeling › The 80-10-10 Masking StrategyOne section · Sections in this chapter4 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Simply replacing all selected tokens with [MASK] creates a mismatch between training and inference. During training, the model sees [MASK] tokens everywhere. During fine-tuning and inference, it never sees them. This discrepancy can hurt transfer performance because the model learns to treat [MASK] as a cue to do its best prediction work, and then loses that cue entirely after pretraining.
BERT addresses this with the 80-10-10 rule. For tokens selected for prediction:
- 80% are replaced with
[MASK] - 10% are replaced with a random token
- 10% are kept unchanged
Each of these three cases serves a distinct purpose, and understanding them reveals the careful reasoning behind the masking design.
The 80% [MASK] case is the primary training signal. Most selected tokens become [MASK], forcing the model to predict the original word from context. This is the core task that drives representation learning. The majority weight ensures that predictions of masked tokens dominate the gradient, keeping the model focused on the primary objective.
The 10% random replacement case trains the model to handle noisy input. When a random token is substituted, the model cannot simply trust that visible tokens are correct. It must learn to represent each token in terms of its identity and how well it fits its context. This is subtle but powerful: even tokens that are not masked must be understood in relation to their neighbors, because any of them might be noisy. As a result, the model's representations become less token-centric and more context-sensitive. Notice that the model still has to output the original token at this position, not the random one, so the loss correctly penalizes the model for believing the random intruder.
The 10% unchanged case closes a different gap. Without this case, the model would learn to only generate meaningful predictions for positions marked by [MASK] and to apply only superficial processing to unmasked positions. By requiring prediction even on unchanged tokens, the model cannot "save" its processing for masked positions. It must maintain rich representations everywhere. In practice, this means that during fine-tuning, when no masking occurs, every token still receives a high-quality contextual representation because the model was trained to produce them uniformly.
All three cases contribute to the loss: the model must predict the original token regardless of what replacement strategy was applied. The loss function does not distinguish between predicting a [MASK] token, recovering from a random substitution, or confirming an unchanged token. They all use the same cross-entropy objective over the vocabulary.
import torch
def apply_mlm_masking(token_ids, vocab_size, mask_token_id, mask_prob=0.15):
"""
Apply BERT-style MLM masking with 80-10-10 strategy.
Args:
token_ids: Original token IDs (batch_size, seq_len)
vocab_size: Size of vocabulary for random replacement
mask_token_id: ID of the [MASK] token
mask_prob: Fraction of tokens to mask (default: 15%)
Returns:
masked_ids: Token IDs with masking applied
labels: Original token IDs at masked positions, -100 elsewhere
"""
labels = token_ids.clone()
masked_ids = token_ids.clone()
# Create probability matrix for masking
probability_matrix = torch.full(token_ids.shape, mask_prob)
# Sample which tokens to mask
masked_indices = torch.bernoulli(probability_matrix).bool()
# Labels are -100 for non-masked tokens (ignored in loss)
labels[~masked_indices] = -100
# 80% of masked tokens -> [MASK]
indices_replaced = (
torch.bernoulli(torch.full(token_ids.shape, 0.8)).bool()
& masked_indices
)
masked_ids[indices_replaced] = mask_token_id
# 10% of masked tokens -> random token
indices_random = (
torch.bernoulli(torch.full(token_ids.shape, 0.5)).bool()
& masked_indices
& ~indices_replaced
)
random_tokens = torch.randint(
vocab_size, token_ids.shape, dtype=token_ids.dtype
)
masked_ids[indices_random] = random_tokens[indices_random]
# Remaining 10% stay unchanged (but still contribute to loss)
return masked_ids, labelsLet's see this masking function in action with a sample sequence:
# Demonstrate the masking on a sample sequence
demo_vocab_size = 30522 # BERT vocab size
demo_mask_token_id = 103 # [MASK] in BERT
# Example sentence (token IDs representing: [CLS] This is a test sentence [SEP])
original = torch.tensor([[101, 2023, 2003, 1037, 3231, 6251, 102]])
masked, labels = apply_mlm_masking(
original, demo_vocab_size, demo_mask_token_id, mask_prob=0.5
)Original tokens: [101, 2023, 2003, 1037, 3231, 6251, 102] Masked tokens: [101, 2023, 103, 1037, 103, 6251, 103] Labels: [-100, -100, 2003, -100, 3231, -100, 102]
The output shows how masking transforms the input. Positions where labels equals -100 are not masked and won't contribute to the loss. Positions with non-negative labels are masked positions where the model must predict the original token. some masked positions show the [MASK] token ID (103), while others show random tokens or remain unchanged. This reflects the 80-10-10 strategy.
The 80-10-10 strategy forces the model to:
- Learn to use context to recover masked tokens (the 80% case)
- Learn representations that tolerate noisy input (the 10% random case)
- Learn that unchanged tokens might still need prediction (the 10% unchanged case)
This last point is subtle but important. By sometimes requiring predictions on unchanged tokens, the model cannot simply "copy" visible tokens to the output. It must use context even for tokens that appear unmodified.
The combined effect of all three cases is that the model's encoder learns to produce high-quality, context-sensitive representations for every token in the sequence, not just for the masked ones. This is exactly what you want for a pretrained encoder that will be fine-tuned on classification, entailment, or retrieval tasks: representations that are rich and stable, as well as independent of any particular masking pattern.