Replaced Token Detection › The RTD Objective

To understand how RTD trains both networks effectively, we need to think about what each network must learn and how to measure its progress. The generator must learn to produce plausible replacements for masked tokens, while the discriminator must learn to detect which tokens have been swapped. These are different tasks requiring different loss functions, yet they must work together in a single training loop.

Let's build up the complete objective by examining each component, understanding why it takes its particular form, and seeing how the pieces combine into an efficient training signal.

The generator faces a familiar task: predict the original token at each masked position. This is identical to masked language modeling. We mask some positions in the input, show the generator the corrupted sequence, and ask it to predict what was hidden.

Why use MLM for the generator? The goal is to produce plausible replacements that will challenge the discriminator. A generator that simply outputs random tokens would be trivial to detect. By training on MLM, the generator learns the statistical patterns of language, producing replacements that fit grammatically and semantically into their contexts. The discriminator then must use syntax and semantics to distinguish these plausible fakes from originals.

Formally, given a set of masked positions $\mathcal{M}$, we minimize the negative log-likelihood of the original tokens:

$$\mathcal{L}_{\text{Gen}} = -\sum_{i \in \mathcal{M}} \log P_G(x_i | \tilde{x})$$

where:

• $\mathcal{L}_{\text{Gen}}$ is the generator loss, identical to standard MLM loss
• $\mathcal{M}$ is the set of masked position indices (typically 15% of positions)
• $x_i$ is the original token at position $i$ that we want to predict
• $\tilde{x}$ is the corrupted input sequence with [MASK] tokens at positions in $\mathcal{M}$
• $P_G(x_i | \tilde{x})$ is the probability the generator assigns to the correct token $x_i$, conditioned on seeing the masked sequence

To understand this formula intuitively, consider what happens as training progresses. Initially, the generator assigns roughly uniform probability across all vocabulary tokens, making $P_G(x_i | \tilde{x}) \approx 1/V$ where $V$ is vocabulary size. The log of this small probability is a large negative number, producing high loss. As the generator learns, it concentrates probability on likely tokens, increasing $P_G(x_i | \tilde{x})$ toward 1. The log approaches zero, and loss decreases.

The summation iterates only over masked positions because those are the only positions where we have a prediction task. Unmasked positions pass through unchanged, contributing nothing to generator learning. This sparsity is the inefficiency that the discriminator's loss will address.

Now we arrive at the key innovation. The discriminator receives a sequence where masked positions have been filled with generator samples, and must classify every token as original or replaced. This binary classification task applies to all positions. This provides the dense training signal that makes RTD efficient.

Think about what the discriminator must learn. For original tokens, it needs to recognize that they fit naturally into their context. For replaced tokens, even plausible ones, it must detect subtle mismatches. Perhaps the replacement "waiter" in "The waiter barked loudly" is grammatically acceptable but semantically wrong. The discriminator learns these nuances by processing the entire sequence and making a decision at each position.

We create the corrupted sequence $\hat{x}$ by sampling from the generator's output distribution at masked positions. The discriminator then outputs a probability for each position, and we compute binary cross-entropy against the ground truth labels:

$$\mathcal{L}_{\text{Disc}} = -\sum_{i=1}^{n} \left[ y_i \log D(\hat{x})_i + (1 - y_i) \log (1 - D(\hat{x})_i) \right]$$

where:

• $\mathcal{L}_{\text{Disc}}$ is the discriminator loss, a sum over all positions
• $n$ is the total sequence length (not just masked positions)
• $y_i$ is the binary label at position $i$ (1 if original, 0 if replaced)
• $\hat{x}$ is the corrupted sequence with generator samples inserted at previously masked positions
• $D(\hat{x})_i$ is the discriminator's predicted probability that position $i$ contains an original token

This binary cross-entropy formula rewards correct predictions at both extremes. Let's trace through what each term contributes:

1. For original tokens ($y_i = 1$): The second term vanishes since $(1 - y_i) = 0$. The loss becomes $-\log D(\hat{x})_i$. When the discriminator correctly predicts high probability (near 1.0), the log is near zero, contributing little loss. When it incorrectly predicts low probability, the log is a large negative number, producing high loss.

2. For replaced tokens ($y_i = 0$): The first term vanishes. The loss becomes $-\log(1 - D(\hat{x})_i)$. When the discriminator correctly predicts low probability (near 0), $(1 - D(\hat{x})_i)$ is near 1, and the log is near zero. When it incorrectly predicts high probability, the loss is large.

The key point is that this loss applies to every position in the sequence. Most positions contain original tokens, so the discriminator primarily learns what "normal" tokens look like in context. A small fraction contain replacements, teaching the discriminator what "wrong" looks like. Both types of positions contribute to learning, giving 6-7x more training signal than MLM's masked-only approach.

With both component losses defined, we need a strategy for combining them into a single objective that trains both networks. The simplest approach is a weighted sum:

$$\mathcal{L}_{\text{RTD}} = \mathcal{L}_{\text{Gen}} + \lambda \cdot \mathcal{L}_{\text{Disc}}$$

where:

• $\mathcal{L}_{\text{RTD}}$ is the total loss used to update both networks
• $\mathcal{L}_{\text{Gen}}$ is the generator's MLM loss (computed only at masked positions)
• $\mathcal{L}_{\text{Disc}}$ is the discriminator's binary classification loss (computed at all positions)
• $\lambda$ is a weighting factor that balances the two losses

Why weight the losses differently? The answer lies in what we ultimately care about. After pretraining, we discard the generator and use only the discriminator for downstream tasks. The generator exists solely to create challenging training examples. From this perspective, generator learning is a means to an end: we want it good enough to produce useful replacements, but not so good that it dominates the training budget.

The ELECTRA paper uses $\lambda = 50$, heavily weighting the discriminator loss. This asymmetry reflects the different roles of the two networks. With this weighting, most of the gradient signal flows to the discriminator, which receives the bulk of the learning capacity. The generator still improves through its share of the gradient, producing increasingly plausible replacements that keep the discriminator challenged.