Replaced Token Detection › RTD Efficiency Advantages
Language AI HandbookAll chapters Replaced Token Detection › RTD Efficiency AdvantagesOne section · Sections in this chapter9 / 17 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The key advantage of RTD is sample efficiency. Let's quantify the difference:
def compare_efficiency(seq_len, mask_prob=0.15):
"""Compare loss signal per token between MLM and RTD."""
# MLM: loss only at masked positions
mlm_loss_positions = int(seq_len * mask_prob)
mlm_efficiency = mlm_loss_positions / seq_len
# RTD: discriminator loss at all positions
rtd_loss_positions = seq_len
rtd_efficiency = rtd_loss_positions / seq_len
efficiency_ratio = rtd_efficiency / mlm_efficiency
return {
"seq_len": seq_len,
"mlm_positions": mlm_loss_positions,
"rtd_positions": rtd_loss_positions,
"mlm_efficiency": mlm_efficiency,
"rtd_efficiency": rtd_efficiency,
"ratio": efficiency_ratio,
}Sample Efficiency Comparison (512 token sequence, 15% masking): MLM loss positions: 76 (14.8% of tokens) RTD loss positions: 512 (100.0% of tokens) RTD efficiency gain: 6.7x more signal per forward pass
This 6-7x efficiency gain is substantial. In practice, ELECTRA achieves comparable results to BERT with roughly 1/4 of the compute. The savings come from learning from every token rather than just masked ones.
It is important to be precise about what "4x more efficient" means. The ELECTRA paper reports results at equal compute budgets, where compute is measured in FLOPs or, equivalently, GPU hours. At the same FLOPs budget, ELECTRA significantly outperforms BERT. This does not mean ELECTRA makes each forward pass 4x cheaper; both models have similar compute per step. Rather, ELECTRA learns faster per step, so it needs fewer steps to reach the same performance. The combination of dense training signal and a binary (rather than vocabulary-scale) output head means each step produces more useful gradient information. Think of it as the difference between studying by reading a textbook passively versus being quizzed on every sentence as you go. The quizzing takes the same time per page, but you learn more per hour because every sentence becomes an active learning opportunity.
