Replaced Token Detection › RTD vs MLM: When to Choose Each
Language AI HandbookAll chapters Replaced Token Detection › RTD vs MLM: When to Choose EachOne section · Sections in this chapter12 / 17 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Both objectives produce strong pretrained models, but they have different strengths and fit different practical scenarios.
Choose RTD (ELECTRA) when:
- Compute budget is limited.
- You need BERT-level performance with less training.
- Pretraining from scratch rather than using existing checkpoints.
- Sample efficiency matters more than final absolute performance.
Choose MLM (BERT) when:
- You can afford extensive pretraining.
- Using established pretrained checkpoints.
- The ecosystem (fine-tuning code, adapters) assumes MLM architecture.
- Simplicity of single-network training is preferred.
The following table summarizes key differences:
| Aspect | MLM (BERT) | RTD (ELECTRA) |
|---|---|---|
| Loss positions | 15% (masked only) | 100% (all positions) |
| Network count | Single transformer | Generator + discriminator |
| Training efficiency | Baseline | ~4x more efficient |
| Architecture complexity | Simple | Moderate |
| Output head | Vocabulary logits | Binary classifier |
| Downstream model | Full encoder | Discriminator only |
The ecosystem consideration deserves more attention than it typically gets. BERT and its derivatives have been the backbone of NLP fine-tuning for years, so much of the tooling and documentation assumes MLM architectures. Hugging Face Transformers and PEFT, along with many domain-specific tools, were built around BERT-like models. When you use ELECTRA, you are using the same discriminator architecture, so fine-tuning code transfers directly. The generator is discarded after pretraining and plays no role in downstream usage. From the perspective of fine-tuning, an ELECTRA discriminator is functionally identical to a BERT encoder: you load it, add a task-specific head, and train exactly the same way. The difference is only in how the encoder was pretrained.
One important practical difference concerns the [MASK] token. BERT uses [MASK] during pretraining and then never sees it during fine-tuning or inference. This creates a distribution mismatch: the model is trained on sequences containing [MASK] but tested on sequences without it. RTD avoids this entirely. The discriminator is never shown [MASK] tokens. Its input during pretraining looks exactly like natural text (original tokens plus generator replacements), which much more closely matches the distribution of text it will see during fine-tuning. This distribution alignment gives RTD an advantage over MLM pretraining.
For practitioners deciding between BERT and ELECTRA for a new pretraining run, the calculus is usually straightforward. If you have a limited budget (say, under 100 GPU-hours), ELECTRA's efficiency advantage means you will likely get better representations at the same cost. If you need to train a very large model and can afford extensive pretraining (as in RoBERTa-scale runs), the two objectives converge in performance and the simpler MLM objective may be preferable. The sweet spot for RTD is the middle regime: medium-sized models with moderate training budgets, which covers a large fraction of real-world NLP applications.