Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Transformer Block Assembly › Parameter Breakdown by Component

Michael BrenndoerferUpdated June 13, 20254 min read

Language AI HandbookAll chapters

Part 15 of 67Transformer Blocks

Chapter 110Transformer Block Assembly › Parameter Breakdown by Component

Reading section 9 of 16 · Parameter Breakdown by Component · 4 minSection 9 of 16 · 4 min

Section 9 of 16 · 4 min read

Transformer Block Assembly › Parameter Breakdown by ComponentOne section · Sections in this chapter9 / 16 ↓

Sections in this chapter16

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Understanding where parameters live helps with model analysis and optimization. Let's break down a complete model:

In[30]:

Code

def analyze_model_parameters(n_layers, d_model, n_heads, d_ff, vocab_size):
    """
    Analyze parameter distribution in a transformer language model.

    Returns dict with parameter counts for each component.
    """
    analysis = {}

    # Embedding (input and output tied)
    analysis["embedding"] = vocab_size * d_model

    # Per-layer components
    per_layer = {}

    # RMSNorm: 2 per layer (attention + FFN) with d_model params each
    per_layer["norm"] = 2 * d_model

    # Attention: Q, K, V, O projections
    per_layer["attention"] = 4 * d_model * d_model

    # SwiGLU FFN: gate, up, down projections
    per_layer["ffn"] = 3 * d_model * d_ff

    per_layer["total"] = sum(per_layer.values())

    # Total for all layers
    analysis["per_layer"] = per_layer
    analysis["all_layers"] = n_layers * per_layer["total"]

    # Final norm
    analysis["final_norm"] = d_model

    # LM head (often tied with embedding)
    analysis["lm_head"] = 0  # Tied with embedding

    # Total
    analysis["total"] = (
        analysis["embedding"] + analysis["all_layers"] + analysis["final_norm"]
    )

    return analysis


# Analyze a GPT-2 Small-like configuration
analysis = analyze_model_parameters(
    n_layers=12,
    d_model=768,
    n_heads=12,
    d_ff=2048,  # ~2.7x with SwiGLU
    vocab_size=50257,
)

Out[31]:

Console

Parameter Analysis (GPT-2 Small-like configuration)
=======================================================

Configuration:
  Layers: 12, d_model: 768, n_heads: 12, d_ff: 2048
  Vocabulary: 50,257 tokens

Per-layer breakdown:
  Norm        :        1,536 (  0.0%)
  Attention   :    2,359,296 ( 33.3%)
  Ffn         :    4,718,592 ( 66.7%)
  ──────────────────────────────
  Total       :    7,079,424

Model totals:
  Embedding:      38,597,376
  All layers:     84,953,088
  Final norm:            768
  ──────────────────────────────
  Total:         123,551,232

  (~124M parameters)

Out[32]:

Visualization

Donut chart showing embedding, transformer layers, and final norm as fractions of total parameters. — Model-level parameter distribution showing embedding and transformer layers.

Donut chart showing norm, attention, and FFN as fractions of per-layer parameters. — Per-layer parameter distribution showing FFN dominance over attention and normalization.

The analysis reveals that FFN parameters dominate within each transformer block. Let's quantify this for SwiGLU:

Attention parameters: $4 \times d_{\text{model}}^2$ (for QKV and O projections)
SwiGLU FFN parameters: $3 \times d_{\text{model}} \times d_{\text{ff}}$ (for gate and up projections plus the down projection)

With typical SwiGLU expansion ( $d_{\text{ff}} \approx 2.7 \times d_{\text{model}}$ ), the FFN has about $8.1 \times d_{\text{model}}^2$ parameters versus attention's $4 \times d_{\text{model}}^2$ . This is roughly a 2:1 ratio, meaning FFN accounts for about 67% of each block's parameters. At the model level, the embedding layer is also significant: with vocabulary size $V$ , the embedding matrix has $V \times d_{\text{model}}$ parameters, which can exceed a single layer's parameters for large vocabularies.