ALBERT › The Parameter Problem
Language AI HandbookAll chapters ALBERT › The Parameter ProblemOne section · Sections in this chapter2 / 18 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
BERT's parameters concentrate in two places: the embedding layer and the transformer blocks. For BERT-Base with a vocabulary of approximately 30,000 tokens and hidden dimension of 768, the embedding matrix alone contains roughly 23 million parameters. That's about 21% of the model's total 110 million parameters, devoted entirely to a lookup table.
The embedding matrix size follows a simple formula: , where is vocabulary size and is hidden dimension. This means larger vocabularies and wider models both inflate embedding costs linearly.
The transformer layers present a different kind of redundancy. Each of BERT's 12 layers has its own attention weights and feed-forward network, but researchers observed that layers often learn similar transformations. The question ALBERT asks: can we share parameters across layers without losing representation quality?
def analyze_bert_parameters(
vocab_size, hidden_size, num_layers, intermediate_size
):
"""Break down BERT's parameter distribution."""
# Embedding parameters
token_embeddings = vocab_size * hidden_size
position_embeddings = 512 * hidden_size # Max sequence length
segment_embeddings = 2 * hidden_size
embedding_layernorm = 2 * hidden_size
total_embedding = (
token_embeddings
+ position_embeddings
+ segment_embeddings
+ embedding_layernorm
)
# Per-layer transformer parameters
# Self-attention: Q, K, V projections + output projection
attention_params = (
4 * hidden_size * hidden_size + 4 * hidden_size
) # weights + biases
attention_layernorm = 2 * hidden_size
# Feed-forward: two linear layers
ffn_params = (
hidden_size * intermediate_size + intermediate_size
) # First layer
ffn_params += intermediate_size * hidden_size + hidden_size # Second layer
ffn_layernorm = 2 * hidden_size
per_layer = (
attention_params + attention_layernorm + ffn_params + ffn_layernorm
)
total_layers = num_layers * per_layer
# Pooler (for classification)
pooler = hidden_size * hidden_size + hidden_size
total = total_embedding + total_layers + pooler
return {
"embedding": total_embedding,
"per_layer": per_layer,
"all_layers": total_layers,
"pooler": pooler,
"total": total,
"embedding_pct": 100 * total_embedding / total,
"layers_pct": 100 * total_layers / total,
}BERT-Base Parameter Distribution: Embeddings: 23,837,184 (21.8%) Per transformer layer: 7,087,872 All 12 layers: 85,054,464 (77.7%) Total: 109,482,240
Over 20% of parameters sit in embeddings, while the remaining ~80% are spread across transformer layers. ALBERT targets both sources of redundancy with separate techniques.
