Replaced Token Detection › The Generator-Discriminator Setup

RTD uses two networks working together: a small generator that creates replacements and a larger discriminator that detects them. This setup resembles generative adversarial networks (GANs) but with an important difference: training uses maximum likelihood, not adversarial objectives. In a GAN, the generator tries to fool the discriminator, and the discriminator tries to catch the generator, creating an adversarial dynamic that can be unstable and difficult to train. In RTD, both networks optimize their own cross-entropy objectives independently. There is no minimax game, no oscillating losses, no mode collapse. The interaction is cooperative in the sense that a better generator makes a better discriminator, but the two networks do not directly compete in the mathematical sense.

The architecture works as follows. First, we mask some positions in the input sequence, exactly as in MLM. The generator, typically a small transformer, predicts tokens for the masked positions. We sample from the generator's output distribution to create replacement tokens. The discriminator, a larger transformer, receives the corrupted sequence and must identify which tokens were replaced.

What the discriminator sees determines the difficulty of its task. It receives the sequence after replacement, meaning it never sees [MASK] tokens. This is intentional. The goal is to train the discriminator to detect subtle token swaps in otherwise natural text. If the discriminator saw [MASK] tokens, it could take a shortcut: any masked position is automatically suspicious. By filling all positions with real tokens, the discriminator must rely on contextual reasoning rather than positional cues.

The size asymmetry between generator and discriminator is also deliberate. The generator is typically one-quarter to one-third the parameter count of the discriminator. This asymmetry serves two purposes. First, it ensures the generator cannot produce perfect replacements. A generator that always produces the exact correct token would make the discriminator's task trivially easy, because every "replaced" token would be correct and indistinguishable from the original. Some fraction of the generator's samples will differ from the original, and those differences are what the discriminator must learn to detect. Second, concentrating parameters in the discriminator maximizes the quality of the representations used for downstream tasks, since the generator is discarded after pretraining.

The key point is that the generator and discriminator have different roles. The generator only needs to produce plausible replacements, not perfect ones. It can be small, perhaps one-quarter or one-third the size of the discriminator. The discriminator, which will be used for downstream tasks, receives the bulk of the parameters and learns rich representations from the detection task.

Notice that the generator's replacements need not be wrong to be useful. When the generator samples the correct token at a masked position, the discriminator still needs to recognize that position as "original." This happens roughly when the generator's most probable token matches the actual token, which occurs more often as the generator improves. These correct-replacement cases are just as important as the wrong-replacement cases because they teach the discriminator what original tokens look like, preventing it from defaulting to a strategy of flagging every formerly-masked position as replaced.