Masked Language Modeling › Limitations and Impact

Masked language modeling has transformed NLP, but it comes with fundamental limitations that shape its applications. Understanding these constraints is essential for making good architectural choices when building systems.

The inability to generate text is the most significant constraint. MLM models cannot produce coherent sequences token by token. They can fill in blanks and score existing text, but they cannot write. This limitation means MLM is unsuitable for chatbots, story generation, code completion, and other generative applications. The underlying reason is architectural: generation requires sequential conditioning, where each new token depends on previously generated tokens. MLM models are trained to condition on the full sequence simultaneously, so there is no natural way to generate left to right. The distinction between understanding and generation has driven the field toward hybrid architectures that combine MLM-style encoders with CLM-style decoders, as seen in T5 and BART.

The masking mismatch between pretraining and fine-tuning creates subtle issues. During pretraining, 15% of tokens are corrupted, and the model sees [MASK] tokens frequently. During fine-tuning, no tokens are masked, and the model never sees [MASK] again. The 80-10-10 strategy mitigates this by keeping 10% of selected tokens unchanged and 10% random, but the model still encounters [MASK] during 80% of masked positions in pretraining and then never during fine-tuning. Research on continuous masking and better pretraining objectives continues to address this gap. ELECTRA (2020) took a fundamentally different approach: instead of masking tokens, it trains a small generator to replace tokens with plausible-sounding alternatives, then trains a discriminator to detect which tokens were replaced. The discriminator receives a signal at every token position (not just masked ones), addressing both the sample efficiency and the masking mismatch simultaneously.

Sample efficiency is another concern. With only 15% of tokens contributing to the loss, MLM requires more compute than CLM to see the same amount of training signal. RoBERTa compensated by training longer and on more data, but this increases cost. ELECTRA showed that the discriminative pretraining signal (every token is real or replaced) converges to better representations with the same compute budget as BERT. Recent work on efficient pretraining explores higher masking rates, span masking, and alternative objectives, but the fundamental tension between bidirectionality and gradient density has not been resolved.

Scaling behavior is also less favorable than CLM. CLM models benefit enormously from scale: larger models with more data consistently produce better generation quality, and many of the most impressive capabilities of GPT-4 and similar models emerge only at large scale. MLM models also improve with scale, but the gains on understanding benchmarks plateau earlier, and the most useful MLM models tend to be in the hundreds-of-millions range rather than the hundreds-of-billions range. The reasons for this difference are still being studied, but it likely relates to the sparse gradient signal and the discriminative nature of MLM pretraining.

Despite these limitations, MLM enabled capabilities that were previously difficult to achieve. BERT's bidirectional representations set new state-of-the-art results on eleven NLP benchmarks when released, including GLUE and SQuAD, as well as CoNLL NER. The pretrain-then-fine-tune paradigm it established remains the dominant approach for understanding tasks. Sentence embeddings from MLM models power semantic search, document clustering, and similarity computations across the industry. Models like Sentence-BERT and instructor-xl, both built on MLM foundations, enable retrieval systems that understand meaning rather than just matching keywords. The insight that bidirectional context improves understanding has influenced the design of virtually every encoder since, and the MLM objective continues to be used in domain-specific pretraining for scientific and medical, plus legal, text.