LLaMA Architecture › Worked Example: Counting Parameters in LLaMA 7B

To solidify your understanding of the architecture, let's trace through where the 7 billion parameters in LLaMA 7B live. This exercise connects the abstract configuration numbers to concrete matrix shapes and gives you an intuition for which components dominate model size.

The LLaMA 7B configuration has $d = 4096$, $n_{\text{layers}} = 32$, $d_{\text{ff}} = 11008$, $n_{\text{heads}} = 32$, and $\text{vocab} = 32000$.

Embedding layer: The token embedding matrix has shape $(\text{vocab\_size}, d) = (32000, 4096)$, contributing $32000 \times 4096 = 131{,}072{,}000$ parameters, roughly 131 million. In LLaMA, the output projection (the matrix that maps the final hidden state back to vocabulary logits) shares weights with the embedding matrix, so this counts only once.

Per-layer parameters: Each of the 32 transformer blocks contains:

• Attention: four projection matrices of shape $(d, d) = (4096, 4096)$, covering QKV plus the output projection. Total per layer: $4 \times 4096^2 \approx 67.1$ million.
• FFN (SwiGLU): three matrices. The gate and up projections have shape $(d, d_{\text{ff}}) = (4096, 11008)$ and the down projection has shape $(d_{\text{ff}}, d) = (11008, 4096)$. Total per layer: $2 \times 4096 \times 11008 + 11008 \times 4096 \approx 135.2$ million.
• RMSNorm: two small parameter vectors of length $d = 4096$ each, one before attention and one before FFN. These add about 8K parameters per layer, negligible at this scale.

Total per layer: approximately $67.1 + 135.2 = 202.3$ million. Across 32 layers: $32 \times 202.3 \approx 6{,}474$ million.

Final normalization: One RMSNorm of size $d = 4096$, negligible.

Total: roughly $131 + 6474 \approx 6{,}605$ million. The gap from 7B to this estimate arises partly from rounding and partly from the fact that "7B" is an approximate label. The actual parameter count for LLaMA 7B is closer to 6.7 billion, consistent with our calculation.

The key observation from this exercise: the feed-forward network contains approximately twice as many parameters as the attention mechanism per layer. The three SwiGLU matrices collectively dwarf the four attention projection matrices. This matters for inference optimization: techniques that quantize or prune the FFN layers have outsized impact on total model size.