ELECTRA › ELECTRA Scaling
Language AI HandbookAll chapters ELECTRA › ELECTRA ScalingOne section · Sections in this chapter10 / 18 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
ELECTRA had a Small configuration, plus Base and Large releases. The configurations follow BERT's patterns but with the addition of a proportionally-sized generator.
electra_configs = {
"ELECTRA-Small": {
"disc_hidden": 256,
"disc_layers": 12,
"disc_heads": 4,
"disc_intermediate": 1024,
"gen_hidden": 256,
"gen_layers": 12,
"gen_heads": 4,
"gen_intermediate": 1024,
"gen_fraction": 1.0, # Same size as discriminator
},
"ELECTRA-Base": {
"disc_hidden": 768,
"disc_layers": 12,
"disc_heads": 12,
"disc_intermediate": 3072,
"gen_hidden": 256,
"gen_layers": 12,
"gen_heads": 4,
"gen_intermediate": 1024,
"gen_fraction": 1 / 3,
},
"ELECTRA-Large": {
"disc_hidden": 1024,
"disc_layers": 24,
"disc_heads": 16,
"disc_intermediate": 4096,
"gen_hidden": 256,
"gen_layers": 24,
"gen_heads": 4,
"gen_intermediate": 1024,
"gen_fraction": 1 / 4,
},
}
def estimate_electra_params(config, vocab_size=30522):
"""Estimate parameters for ELECTRA configuration."""
# Discriminator parameters
disc_h = config["disc_hidden"]
disc_l = config["disc_layers"]
disc_i = config["disc_intermediate"]
disc_embedding = vocab_size * disc_h + 512 * disc_h # Token + position
disc_attention = disc_l * (4 * disc_h * disc_h + 4 * disc_h)
disc_ffn = disc_l * (disc_h * disc_i + disc_i + disc_i * disc_h + disc_h)
disc_layernorm = disc_l * 4 * disc_h
disc_head = disc_h + 1 # Binary classifier
disc_total = (
disc_embedding + disc_attention + disc_ffn + disc_layernorm + disc_head
)
# Generator parameters
gen_h = config["gen_hidden"]
gen_l = config["gen_layers"]
gen_i = config["gen_intermediate"]
gen_embedding = vocab_size * gen_h + 512 * gen_h
gen_attention = gen_l * (4 * gen_h * gen_h + 4 * gen_h)
gen_ffn = gen_l * (gen_h * gen_i + gen_i + gen_i * gen_h + gen_h)
gen_layernorm = gen_l * 4 * gen_h
gen_head = gen_h * vocab_size + vocab_size # MLM head
gen_total = (
gen_embedding + gen_attention + gen_ffn + gen_layernorm + gen_head
)
return {
"discriminator": disc_total,
"generator": gen_total,
"total": disc_total + gen_total,
"disc_only": disc_total, # What gets used for fine-tuning
}ELECTRA Model Configurations: -------------------------------------------------------------------------------- Model Disc Hidden Gen Hidden Disc Params Total Params -------------------------------------------------------------------------------- ELECTRA-Small 256 256 17.4M 42.7M ELECTRA-Base 768 256 108.9M 134.2M ELECTRA-Large 1024 256 334.1M 368.8M
The discriminator size scales from ~14M parameters (Small) to ~335M (Large), matching BERT's configurations. The generator stays fixed at 256 hidden dimensions across all sizes, contributing only ~16M additional parameters. For downstream tasks, only the discriminator parameters matter. The generator is discarded after pre-training.
Notice that the generator is always small (256 hidden size) regardless of discriminator size. This is intentional. A small generator produces imperfect replacements that are challenging but not impossible for the discriminator. The generator parameters are discarded after pre-training; only the discriminator is used for downstream tasks.
