Architecture Hyperparameters › Total Parameter Calculation
Language AI HandbookAll chapters Architecture Hyperparameters › Total Parameter CalculationOne section · Sections in this chapter6 / 10 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
We've now examined each component of a transformer: embeddings that map tokens to vectors, attention mechanisms that mix information across positions, and feed-forward networks that transform each position independently. To design architectures or understand published models, we need to combine these pieces into a complete parameter count.
This matters for several practical reasons:
-
Hardware planning: GPU memory limits how many parameters you can train. A 70B parameter model in FP16 requires 140GB just for weights, before accounting for optimizer states, activations, or gradients.
-
Cost estimation: Training compute scales with parameter count. Knowing your model's size helps estimate training time and cost.
-
Architecture verification: When implementing published models, parameter counting helps verify your implementation matches the original.
-
Design iteration: When exploring architectures, quick parameter estimates help evaluate trade-offs without running experiments.
Let's build a parameter calculator that breaks down contributions from each component.
def full_parameter_count(
V, # Vocabulary size
L, # Number of layers
d_model, # Model dimension
n_heads, # Number of attention heads
d_ff, # Feed-forward dimension
n_kv_heads=None, # Number of KV heads (None = standard MHA)
ffn_type="standard", # "standard" or "swiglu"
tie_embeddings=True, # Whether output embeddings are tied to input
include_biases=False, # Whether to count bias parameters
):
"""
Calculate total parameters for a transformer language model.
Returns detailed breakdown by component.
"""
if n_kv_heads is None:
n_kv_heads = n_heads
d_head = d_model // n_heads
params = {}
# Embedding layer
params["embedding"] = V * d_model
# Attention parameters per layer
q_params = n_heads * d_head * d_model
k_params = n_kv_heads * d_head * d_model
v_params = n_kv_heads * d_head * d_model
o_params = n_heads * d_head * d_model
if include_biases:
q_params += n_heads * d_head
k_params += n_kv_heads * d_head
v_params += n_kv_heads * d_head
o_params += d_model
params["attention_per_layer"] = q_params + k_params + v_params + o_params
# FFN parameters per layer
if ffn_type == "standard":
ffn_weights = 2 * d_model * d_ff
ffn_biases = d_ff + d_model if include_biases else 0
else: # SwiGLU
ffn_weights = 3 * d_model * d_ff
ffn_biases = 0 # SwiGLU typically doesn't use biases
params["ffn_per_layer"] = ffn_weights + ffn_biases
# Layer norms (2 per layer, each with d_model scale params)
# RMSNorm has d_model params, LayerNorm has 2*d_model (scale + shift)
params["norm_per_layer"] = 2 * d_model # Assuming RMSNorm
# Total per layer
params["per_layer"] = (
params["attention_per_layer"]
+ params["ffn_per_layer"]
+ params["norm_per_layer"]
)
# All layers
params["all_layers"] = L * params["per_layer"]
# Final layer norm
params["final_norm"] = d_model
# Output projection
if tie_embeddings:
params["lm_head"] = 0
else:
params["lm_head"] = d_model * V
if include_biases:
params["lm_head"] += V
# Total
params["total"] = (
params["embedding"]
+ params["all_layers"]
+ params["final_norm"]
+ params["lm_head"]
)
# Additional metadata
params["metadata"] = {
"attention_fraction": params["attention_per_layer"]
/ params["per_layer"],
"ffn_fraction": params["ffn_per_layer"] / params["per_layer"],
"embedding_fraction": params["embedding"] / params["total"],
"layers_fraction": params["all_layers"] / params["total"],
}
return paramsLet's verify our calculator against known model sizes:
# Verify against published model sizes
verification_models = {
"GPT-2 Small (124M)": {
"V": 50257,
"L": 12,
"d_model": 768,
"n_heads": 12,
"d_ff": 3072,
"ffn_type": "standard",
"tie_embeddings": True,
},
"GPT-2 XL (1.5B)": {
"V": 50257,
"L": 48,
"d_model": 1600,
"n_heads": 25,
"d_ff": 6400,
"ffn_type": "standard",
"tie_embeddings": True,
},
"LLaMA-7B": {
"V": 32000,
"L": 32,
"d_model": 4096,
"n_heads": 32,
"d_ff": 11008,
"ffn_type": "swiglu",
"tie_embeddings": True,
},
"LLaMA-2 70B": {
"V": 32000,
"L": 80,
"d_model": 8192,
"n_heads": 64,
"n_kv_heads": 8, # GQA with 8 KV heads
"d_ff": 28672,
"ffn_type": "swiglu",
"tie_embeddings": True,
},
}
verification_results = {}
for name, config in verification_models.items():
result = full_parameter_count(**config)
verification_results[name] = resultParameter count verification: Model Calculated Embedding % Layers % ----------------------------------------------------------------- GPT-2 Small (124M) 123.55M 31.2% 68.8% GPT-2 XL (1.5B) 1.56B 5.2% 94.8% LLaMA-7B 6.61B 2.0% 98.0% LLaMA-2 70B 68.71B 0.4% 99.6%
Our calculations align with published model sizes. Note how the embedding fraction decreases dramatically as models scale: for GPT-2 Small, embeddings represent a significant fraction of parameters, but for LLaMA-70B, the transformer layers dominate overwhelmingly. This is why larger models are more parameter-efficient: the fixed embedding cost amortizes over more layer parameters.
The visualization confirms what the numbers suggested: as models grow, FFN parameters increasingly dominate. The FFN is the workhorse of parameter storage, containing roughly twice the parameters of attention in each layer. This explains why techniques like mixture-of-experts (MoE), which primarily target FFN layers, can dramatically increase effective capacity.