Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

ELECTRA › The Inefficiency of Masked Language Modeling

Michael BrenndoerferUpdated July 25, 20253 min read

Language AI HandbookAll chapters

Part 24 of 67BERT and Variants

Chapter 175ELECTRA › The Inefficiency of Masked Language Modeling

Reading section 2 of 18 · The Inefficiency of Masked Language Modeling · 3 minSection 2 of 18 · 3 min

Section 2 of 18 · 3 min read

ELECTRA › The Inefficiency of Masked Language ModelingOne section · Sections in this chapter2 / 18 ↓

Sections in this chapter18

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let's quantify what BERT wastes. For a 512-token sequence with 15% masking, only 77 tokens contribute to the MLM loss. The other 435 tokens pass through 12 transformer layers, consuming attention computations and memory, but generate no gradient signal. They're essentially free riders.

In[4]:

Code

def compute_mlm_efficiency(seq_len, mask_prob):
    """Calculate what fraction of compute contributes to learning."""
    masked_tokens = int(seq_len * mask_prob)
    unmasked_tokens = seq_len - masked_tokens

    # All tokens consume forward pass compute
    forward_compute = seq_len

    # Only masked tokens contribute to loss
    learning_tokens = masked_tokens

    efficiency = learning_tokens / forward_compute

    return {
        "seq_len": seq_len,
        "masked_tokens": masked_tokens,
        "unmasked_tokens": unmasked_tokens,
        "efficiency": efficiency,
    }

Out[5]:

Console

BERT MLM Efficiency Analysis:
  Sequence length: 512 tokens
  Masked tokens: 76 (contribute to loss)
  Unmasked tokens: 436 (no learning signal)
  Training efficiency: 14.8%

Out[6]:

Visualization

Pie chart showing 15% of tokens contribute to learning while 85% provide no gradient signal. — Token contribution breakdown in BERT's masked language modeling. Only 15% of tokens in each batch contribute to the loss function, while the remaining 85% consume compute without providing learning signal.

Only 15% of tokens contribute to learning. This is inherent to the MLM objective: predicting from a large vocabulary is hard, so we can only mask a small fraction without destroying context. Mask too many tokens and the task becomes impossible.

ELECTRA sidesteps this limitation by changing the task entirely. Instead of predicting masked tokens (a 30,000-way classification problem), the model classifies each token as original or replaced (a binary classification problem). Binary classification per token means every position can contribute to the loss without overwhelming the model.

Replaced Token Detection (RTD)

A pre-training objective where the model predicts, for each token in a sequence, whether it is the original token or a replacement generated by a separate model. Unlike MLM's vocabulary-sized softmax, RTD uses a simple sigmoid classifier, enabling learning from every token position.