Replaced Token Detection › Worked Example: One Forward Pass Through RTD

To make the abstract concrete, let's trace through a complete RTD forward pass step by step, computing losses by hand on a toy example. This will make the loss formulas feel real before we see them in code.

The sentence: "The scientist discovered a cure."

Step 1: Masking. We mask 15% of positions. For a 5-token sequence, this typically means 1 position. Suppose we mask position 1 ("scientist"), producing the sequence: "The [MASK] discovered a cure."

Step 2: Generator forward pass. The generator processes this masked sequence and produces a probability distribution over the vocabulary at each position. At position 1 (the masked position), suppose the generator assigns the following probabilities to the top candidates:

• "researcher": 0.31
• "doctor": 0.22
• "scientist": 0.18
• "professor": 0.14
• (other tokens): 0.15 combined

The generator loss at this position is $-\log(0.18) \approx 1.71$, since "scientist" (probability 0.18) is the correct token. This loss signal pushes the generator to assign higher probability to "scientist."

Step 3: Sampling. We sample from the generator's distribution at the masked position. Suppose we draw "researcher" (probability 0.31). The corrupted sequence becomes: "The researcher discovered a cure."

Step 4: Label creation. We compare the corrupted sequence with the original:

• Position 0: "The" unchanged. Label = 1 (original).
• Position 1: "researcher" replaced "scientist". Label = 0 (replaced).
• Position 2: "discovered" unchanged. Label = 1 (original).
• Position 3: "a" unchanged. Label = 1 (original).
• Position 4: "cure" unchanged. Label = 1 (original).

Step 5: Discriminator forward pass. The discriminator processes the corrupted sequence "The researcher discovered a cure" and produces a probability for each position. Suppose it outputs these "is original" probabilities: [0.92, 0.34, 0.88, 0.95, 0.91].

Step 6: Discriminator loss. We compute binary cross-entropy at each position using the ground truth labels:

$$\mathcal{L}_{\text{Disc}} = \sum_{i=0}^{4} \left[ -y_i \log(p_i) - (1 - y_i) \log(1 - p_i) \right]$$

Working through position by position:

• Position 0 (original, $y=1$, $p=0.92$): $-\log(0.92) \approx 0.083$
• Position 1 (replaced, $y=0$, $p=0.34$): $-\log(1 - 0.34) = -\log(0.66) \approx 0.415$
• Position 2 (original, $y=1$, $p=0.88$): $-\log(0.88) \approx 0.128$
• Position 3 (original, $y=1$, $p=0.95$): $-\log(0.95) \approx 0.051$
• Position 4 (original, $y=1$, $p=0.91$): $-\log(0.91) \approx 0.094$

Total discriminator loss: $0.083 + 0.415 + 0.128 + 0.051 + 0.094 = 0.771$.

Notice that the discriminator got position 1 "wrong" in the sense that it assigned probability 0.34, which means it thought there was a 34% chance "researcher" was original. Ideally, this probability would be near 0. The loss at position 1 (0.415) is the largest single contribution, pulling the discriminator toward assigning lower probability at positions where semantically plausible but contextually imperfect replacements appear.

Step 7: Combined loss. With $\lambda = 50$:

$$\mathcal{L}_{\text{RTD}} = 1.71 + 50 \times 0.771 = 1.71 + 38.55 = 40.26$$

The discriminator component ($38.55$) dwarfs the generator component ($1.71$), exactly as intended. The optimizer updates both networks, but the gradient flowing from the combined loss is overwhelmingly shaped by the discriminator's needs.

This one example illustrates something important: the training signal at position 1 is what teaches the discriminator to notice when a semantically related but contextually wrong word appears. "Researcher" is plausible in isolation. In the phrase "the researcher discovered a cure," it is even grammatically correct. Yet something about the surrounding context, perhaps the formal scientific register of "discovered a cure," might make "scientist" slightly more natural. The discriminator learns to pick up on these subtle coherence signals by being forced to classify every position, including those that were not masked.