Replaced Token Detection › Summary

Replaced token detection reformulates pretraining as a detection problem rather than a generation problem. This reformulation has one simple motivation (eliminate the 85% of wasted forward pass positions in MLM) and one key insight (detection is easier than generation, so the model can learn from every position without requiring the capacity needed for exact token prediction). Together, these produce a pretraining objective that trains encoders more efficiently than MLM across a wide range of compute budgets.

The key concepts from this chapter are:

• Generator-discriminator architecture uses a small network to produce replacement tokens and a larger network to detect them, with only the discriminator used for downstream tasks.
• Detection vs. generation is easier because binary classification requires less capacity than vocabulary prediction, enabling learning from every position.
• Sample efficiency improves by 6-7x because the discriminator loss applies to all positions rather than only the 15% that would be masked in MLM.
• Generator sizing at 1/4 to 1/3 of discriminator size balances challenging replacements against parameter efficiency.
• Weight sharing between generator and discriminator embeddings reduces parameters while maintaining compatible representations.
• Training dynamics create complementary learning where better generators produce harder replacements that train better discriminators.
• No adversarial training is needed: both networks optimize independent maximum likelihood objectives, making RTD more stable to train than GAN-based approaches despite the generator-discriminator framing.

ELECTRA's success with RTD demonstrated that the MLM paradigm, despite its widespread adoption, was not the most efficient path to strong language representations. The core ideas, assigning learning tasks to every input position and using detection rather than generation as the pretraining signal, have influenced subsequent work on efficient pretraining and continue to be relevant as the field explores better use of limited compute budgets.

The next chapter explores denoising objectives, a family of pretraining tasks that corrupt inputs in various ways and train models to reconstruct them.