ELECTRA › Why No Adversarial Training?

Michael BrenndoerferUpdated July 25, 20253 min read
Language AI HandbookAll chapters
Part 24 of 67BERT and Variants
Chapter 175ELECTRA › Why No Adversarial Training?
Reading section 8 of 18 · Why No Adversarial Training? · 3 minSection 8 of 18 · 3 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 175 of 513

Section 8 of 18 · 3 min read

ELECTRA › Why No Adversarial Training?One section · Sections in this chapter8 / 18 ↓
Sections in this chapter18
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

You might wonder why ELECTRA doesn't use adversarial training like a GAN. The generator could try to fool the discriminator, making training competitive rather than cooperative. The ELECTRA authors experimented with this and found it hurt performance.

The problem is the discreteness of text. GANs work well for continuous outputs (images, audio) where small gradient-based adjustments smoothly improve quality. Text is discrete: you either sample token 7842 or you don't. There's no gradient to flow back from the discriminator's judgment of that specific token choice.

Techniques like REINFORCE can estimate gradients through discrete sampling, but they introduce high variance. In ELECTRA's experiments, adversarial training with REINFORCE produced unstable training and worse results. The simple cooperative approach, where both networks minimize their own losses, proved more effective.

Out[22]:
Visualization
Diagram comparing cooperative and adversarial training paradigms with arrows showing gradient flow direction.
Cooperative versus adversarial training in ELECTRA. In cooperative training (used), both networks minimize their own losses. In adversarial training (not used), the generator tries to maximize the discriminator's loss. Cooperative training is more stable for discrete outputs like text.