Replaced Token Detection › Generator Size and Training Dynamics

How big should the generator be relative to the discriminator? Too small, and it produces random, easily-detected replacements. Too large, and it wastes parameters that could go to the discriminator, which is the network used for downstream tasks.

The ELECTRA paper found that generator size between 1/4 and 1/2 of the discriminator works best. The sweet spot balances challenging replacements against parameter efficiency. For ELECTRA-Base (which aims to match BERT-Base with 110M parameters), the discriminator uses 12 layers with hidden dimension 256, while the generator uses 12 layers with hidden dimension 64. The layer count is the same, but the hidden dimension is much smaller, resulting in a generator with roughly one-quarter the parameters. For ELECTRA-Large, the discriminator matches BERT-Large (24 layers, hidden dimension 1024) while the generator uses hidden dimension 256.

The intuition behind the sweet spot is straightforward. A generator with essentially zero capacity (say, a single embedding lookup) would produce replacements based only on unigram frequency, the most common words in the language. These replacements would be detected trivially because they ignore context entirely. A generator with the same capacity as the discriminator would produce near-perfect replacements, making the discrimination task nearly impossible. Neither extreme leads to useful discriminator training. The optimal generator is one that understands enough of the language to create contextually plausible replacements but not so much that it can consistently reproduce the exact original token.

In practice, this means you should resist the temptation to make the generator too powerful when setting up RTD training from scratch. If you have a total parameter budget of, say, 100M, allocating 75M to the discriminator and 25M to the generator will outperform an even 50/50 split. The generator is a means to an end: its job is to create training examples for the discriminator, and once pretraining is complete, it is discarded entirely.

The training dynamics create an interesting interplay. As the generator improves, it produces harder replacements. This forces the discriminator to encode finer contextual distinctions. But if the generator becomes too good, detection becomes impossible, and learning stalls. The asymmetric sizing naturally prevents this collapse: the smaller generator cannot outpace the larger discriminator.

This self-regulating property is one of RTD's practical advantages over GAN-based approaches. In a GAN, you must carefully balance generator and discriminator learning rates to prevent one from dominating. In RTD, the parameter asymmetry does this automatically. The generator is trained on the same number of steps as the discriminator but with fewer parameters, giving it an inherently limited ceiling. It will improve, but it will improve slowly enough that the discriminator can keep pace.