ELECTRA › The Generator-Discriminator Architecture

Michael BrenndoerferUpdated July 25, 20253 min read
Language AI HandbookAll chapters
Part 24 of 67BERT and Variants
Chapter 175ELECTRA › The Generator-Discriminator Architecture
Reading section 3 of 18 · The Generator-Discriminator Architecture · 3 minSection 3 of 18 · 3 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 175 of 513

Section 3 of 18 · 3 min read

ELECTRA › The Generator-Discriminator ArchitectureOne section · Sections in this chapter3 / 18 ↓
Sections in this chapter18
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

ELECTRA uses two transformer networks: a generator and a discriminator. The generator is a small masked language model that proposes plausible replacements for masked tokens. The discriminator is a larger network that learns to detect which tokens were replaced. Despite the naming, this is not adversarial training in the GAN sense. Both networks are trained jointly to minimize their respective losses.

Out[7]:
Visualization
Diagram showing input flowing through generator to create corrupted sequence, then through discriminator which outputs original/replaced predictions for each position.
ELECTRA's two-network architecture. The generator (small transformer) fills in masked positions with plausible tokens. The discriminator (large transformer) classifies each token as original or replaced. Both networks train simultaneously, but only the discriminator is used for downstream tasks.

The key insight is that the generator doesn't need to be good, just good enough to fool a naive discriminator occasionally. A small generator (1/4 to 1/3 the discriminator's size) works well because it produces plausible but imperfect replacements. If the generator were perfect, every replacement would match the original, and the discriminator would learn nothing. If the generator were random, the replacements would be obviously wrong, making discrimination trivial. The sweet spot is a generator that produces contextually appropriate tokens that are sometimes, but not always, correct.