Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

ELECTRA › Sample Efficiency

Michael BrenndoerferUpdated July 25, 20254 min read

Language AI HandbookAll chapters

Part 24 of 67BERT and Variants

Chapter 175ELECTRA › Sample Efficiency

Reading section 9 of 18 · Sample Efficiency · 4 minSection 9 of 18 · 4 min

Section 9 of 18 · 4 min read

ELECTRA › Sample EfficiencyOne section · Sections in this chapter9 / 18 ↓

Sections in this chapter18

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

ELECTRA's main contribution is sample efficiency: it achieves the same performance as BERT with far less compute. The original paper demonstrated this across multiple scales.

At the small scale (comparable to BERT-Small with 14M parameters), ELECTRA trained for 1/4 of BERT's training steps matched BERT's full-training performance. At the base scale (110M parameters), ELECTRA trained with 1/4 the compute exceeded BERT's performance. At the large scale (335M parameters), ELECTRA achieved new state-of-the-art results on GLUE.

Out[23]:

Visualization

Line plot showing GLUE score versus compute FLOPs for BERT and ELECTRA, with ELECTRA consistently higher. — ELECTRA's sample efficiency compared to BERT. At each compute budget, ELECTRA achieves higher GLUE scores. With 1/4 the compute, ELECTRA matches BERT-Base performance. With equal compute, ELECTRA exceeds BERT by several points.

Why is ELECTRA so efficient? Three factors contribute:

Learning from all tokens: RTD provides signal for every position, not just 15%. This roughly 6.7x increase in gradient information per batch directly translates to faster learning.
Easier task per token: Binary classification is simpler than vocabulary prediction. The discriminator can focus its capacity on understanding context rather than memorizing rare vocabulary items.
Generator bootstrapping: The small generator quickly learns to produce plausible replacements. This provides meaningful training signal to the discriminator from early in training.

In[24]:

Code

def compare_learning_signal():
    """Compare gradient information per batch between MLM and RTD."""
    seq_len = 512
    mask_prob = 0.15
    vocab_size = 30522

    # MLM: gradient only from masked positions
    mlm_positions = int(seq_len * mask_prob)
    mlm_bits_per_position = np.log2(vocab_size)  # ~15 bits for 30k vocab
    mlm_total_bits = mlm_positions * mlm_bits_per_position

    # RTD: gradient from all positions
    rtd_positions = seq_len
    rtd_bits_per_position = 1  # binary classification
    rtd_total_bits = rtd_positions * rtd_bits_per_position

    return {
        "mlm_positions": mlm_positions,
        "mlm_bits_per_pos": mlm_bits_per_position,
        "mlm_total_bits": mlm_total_bits,
        "rtd_positions": rtd_positions,
        "rtd_bits_per_pos": rtd_bits_per_position,
        "rtd_total_bits": rtd_total_bits,
        "position_ratio": rtd_positions / mlm_positions,
    }

Out[25]:

Console

Learning Signal Comparison (per 512-token sequence):

MLM (BERT):
  Positions with gradient: 76
  Classification complexity: 14.9 bits (30k-way)
  Total information: 1132 bits

RTD (ELECTRA):
  Positions with gradient: 512
  Classification complexity: 1 bit (binary)
  Total information: 512 bits

Position coverage improvement: 6.7x

ELECTRA provides gradient signal at 6.7× more positions than MLM. While the per-position information content differs (15 bits for vocabulary prediction versus 1 bit for binary classification), the increased coverage more than compensates. The model receives learning signal from every token, including positions that remain unmasked.

The comparison isn't quite apples-to-apples because MLM's vocabulary-sized prediction is harder per position than RTD's binary classification. But the positions matter more than the per-position complexity: learning about context from 512 positions beats learning about vocabulary from 77 positions. The model needs to understand context to succeed at either task, and more positions means more context learning.