Replaced Token Detection › Replaced Token Detection
Language AI HandbookAll chapters Replaced Token Detection › Replaced Token DetectionOne section · Sections in this chapter1 / 17 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Masked language modeling wastes compute. When BERT masks 15% of tokens and predicts only those positions, 85% of the forward pass produces no training signal. The model processes the entire sequence but learns from a small fraction of it. For expensive pretraining runs consuming millions of GPU hours, this inefficiency is costly. When you multiply that waste across millions of training steps, the cumulative cost in both time and money becomes substantial.
Think of it this way: every time BERT reads a 512-token sequence, it performs a full transformer forward pass over all 512 positions. The attention layers compare every token against every other token. The feed-forward layers transform each position's representation. Yet only about 77 of those positions, the ones that were masked, produce any gradient signal. The other 435 positions are passengers, processed but not learned from. This is the inefficiency that Replaced Token Detection (RTD) was designed to eliminate.
Replaced Token Detection solves this problem with a different formulation. Instead of predicting masked tokens, the model classifies every token as either original or replaced. A small "generator" network produces plausible replacements for masked positions, and a larger "discriminator" network learns to detect which tokens have been swapped. Every position contributes to the loss, making training much more sample-efficient. This shift increases how much training signal each sequence provides.
This approach, introduced in the ELECTRA paper in 2020, achieves BERT-level performance with a fraction of the compute. The key insight is that detection is easier than generation: distinguishing real from fake requires less capacity than predicting the exact original token, allowing the model to learn from every position rather than just masked ones. The paper demonstrated that with roughly one-quarter of BERT's pretraining budget, ELECTRA could match or exceed BERT's performance on the GLUE benchmark. This result sent a clear signal to the NLP community: the MLM paradigm, despite its success, had not found the most efficient path to strong representations.
BERT's introduction in 2018 established masked language modeling as the dominant pretraining paradigm, but it also set a worrying precedent: competitive pretraining required enormous compute. Google's BERT-Large required 4 days on 64 TPU chips. The subsequent GPT and RoBERTa models pushed these requirements even higher. By 2019, researchers were asking whether the MLM objective itself was the bottleneck. Kevin Clark and colleagues at Stanford and Google Brain answered this question with ELECTRA (Efficiently Learning an Encoder that Classifies Token Replacements Accurately), published at ICLR 2020. Their central observation, that the 15% masking rate in MLM is an inherent waste, led to the replaced token detection objective. ELECTRA-Small outperformed GPT on GLUE while using less compute than BERT-Base. The paper helped catalyze what became known as the "efficiency turn" in NLP research, sparking interest in compute-optimal training that would later influence scaling law research.
To understand why RTD is more efficient, it helps to contrast the two objectives at an intuitive level. MLM asks a hard question at a small number of positions: "What exact word was hidden here?" The model must consult its vocabulary of 30,000 tokens and assign probability across all of them. RTD asks an easier question at every position: "Does this word belong here, or was it swapped in?" The model only needs to output a single probability per token. Easier questions answered everywhere can teach more than harder questions answered rarely, and that tradeoff is the heart of RTD.