ALBERT › ALBERT Model Configurations
Language AI HandbookAll chapters ALBERT › ALBERT Model ConfigurationsOne section · Sections in this chapter10 / 18 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
albert_configs = {
"ALBERT-Base": {
"embedding_dim": 128,
"hidden_dim": 768,
"num_heads": 12,
"intermediate_dim": 3072,
"num_layers": 12,
},
"ALBERT-Large": {
"embedding_dim": 128,
"hidden_dim": 1024,
"num_heads": 16,
"intermediate_dim": 4096,
"num_layers": 24,
},
"ALBERT-xLarge": {
"embedding_dim": 128,
"hidden_dim": 2048,
"num_heads": 16,
"intermediate_dim": 8192,
"num_layers": 24,
},
"ALBERT-xxLarge": {
"embedding_dim": 128,
"hidden_dim": 4096,
"num_heads": 64,
"intermediate_dim": 16384,
"num_layers": 12,
},
}
def estimate_albert_params(config, vocab_size=30522):
"""Estimate parameter count for an ALBERT configuration."""
e = config["embedding_dim"]
h = config["hidden_dim"]
i = config["intermediate_dim"]
heads = config["num_heads"]
# Factorized embeddings
embedding_params = (
vocab_size * e + 512 * e + 2 * e
) # Token, position, segment
projection_params = e * h # Embed projection
# Shared layer (only counted once)
attention_params = 4 * h * h + 4 * h # Q, K, V, O
ffn_params = h * i + i + i * h + h # Two layers with biases
layer_params = attention_params + ffn_params + 4 * h # Plus LayerNorms
# Heads
mlm_head = (
h * e + e + 2 * e + vocab_size * e
) # Dense + norm + decoder (tied)
sop_head = h * 2
pooler = h * h + h
total = (
embedding_params
+ projection_params
+ layer_params
+ mlm_head
+ sop_head
+ pooler
)
return totalALBERT Model Configurations: --------------------------------------------------------------------------- Model Hidden Layers Intermediate Est. Params --------------------------------------------------------------------------- ALBERT-Base 768 12 3072 15.8M ALBERT-Large 1024 24 4096 21.8M ALBERT-xLarge 2048 24 8192 63.0M ALBERT-xxLarge 4096 12 16384 227.1M
Notice how parameter counts grow much slower than hidden dimensions. ALBERT-xxLarge has a hidden dimension 5.3x larger than ALBERT-Base (4096 vs 768), but its parameter count is only about 20x larger. With BERT's approach, the parameter ratio would match the hidden dimension ratio squared, since attention parameters scale with .
The largest model, ALBERT-xxLarge, uses only 12 layers but has a massive 4096-dimensional hidden space. This reflects the finding that wider models with shared parameters can match deeper models with unique parameters.
We can visualize how differently parameters scale between BERT's approach and ALBERT's:
