Replaced Token Detection › In Practice: ELECTRA Performance Benchmarks

The efficiency claims in the ELECTRA paper are backed by concrete benchmark results. ELECTRA-Small, trained for 1M steps on the same dataset as BERT, outperforms GPT (a model many times its compute cost) on the GLUE benchmark. ELECTRA-Base matches RoBERTa-Base while using about 25% of its compute. ELECTRA-Large, trained to the same compute budget as RoBERTa-Large, exceeds RoBERTa-Large's performance.

The key takeaway from these results is that ELECTRA's advantage is most pronounced at lower compute budgets. When you can afford to train for a long time (rightward on the x-axis), both objectives eventually converge to similar performance levels. When you have limited compute, the RTD advantage is dramatic. This is why the original ELECTRA paper emphasizes the efficiency framing: it speaks directly to the practitioners who cannot afford the compute budgets of large industrial research labs.