Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

LLaMA Components › Parameter Distribution

Michael BrenndoerferUpdated August 2, 20253 min read

Language AI HandbookAll chapters

Part 29 of 67Modern Decoder Models

Chapter 208LLaMA Components › Parameter Distribution

Reading section 8 of 12 · Parameter Distribution · 3 minSection 8 of 12 · 3 min

Section 8 of 12 · 3 min read

LLaMA Components › Parameter DistributionOne section · Sections in this chapter8 / 12 ↓

Sections in this chapter12

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Understanding where parameters reside helps explain why certain design choices matter. Let's analyze the parameter distribution in a LLaMA block:

In[23]:

Code

def count_block_params(d_model, n_heads, d_ff):
    """Count parameters in each component of a LLaMA block."""
    d_head = d_model // n_heads

    params = {
        "RMSNorm (×2)": 2 * d_model,  # Two norm layers, one scale per dim
        "Q projection": d_model * d_model,
        "K projection": d_model * d_model,
        "V projection": d_model * d_model,
        "O projection": d_model * d_model,
        "FFN W_gate": d_model * d_ff,
        "FFN W_up": d_model * d_ff,
        "FFN W_down": d_ff * d_model,
    }

    return params


# LLaMA 7B proportions
d_model = 4096
n_heads = 32
d_ff = 11008  # Actual LLaMA 7B FFN dimension

params = count_block_params(d_model, n_heads, d_ff)
total = sum(params.values())

Out[24]:

Visualization

Pie chart showing FFN taking 67%, attention 33%, and normalization less than 1%. — Parameter distribution by component. SwiGLU FFN dominates with approximately 67% of parameters.

Bar chart showing parameter counts for each weight matrix in the transformer block. — Detailed parameter breakdown by matrix. Each FFN projection contributes about 45M parameters.

Out[25]:

Console

Parameter Count per Block (LLaMA 7B scale):
--------------------------------------------------
  RMSNorm (×2)        :        8,192 (  0.0%)
  Q projection        :   16,777,216 (  8.3%)
  K projection        :   16,777,216 (  8.3%)
  V projection        :   16,777,216 (  8.3%)
  O projection        :   16,777,216 (  8.3%)
  FFN W_gate          :   45,088,768 ( 22.3%)
  FFN W_up            :   45,088,768 ( 22.3%)
  FFN W_down          :   45,088,768 ( 22.3%)
--------------------------------------------------
  Total               :  202,383,360 (100.0%)

For 32 layers: 6.48B parameters in transformer blocks

The FFN dominates parameter count despite SwiGLU's efficiency improvements. This is because:

The hidden dimension $d_{\text{ff}}$ is large (about 2.7x $d_{\text{model}}$ )
SwiGLU uses three matrices instead of two

However, this investment pays off. The FFN provides the model's primary capacity for storing learned patterns and facts. The attention mechanism, while containing fewer parameters, handles the important task of routing information between positions.