Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

LLaMA Architecture › Model Architecture

Michael BrenndoerferUpdated August 8, 202515 min read

Language AI HandbookAll chapters

Part 29 of 67Modern Decoder Models

Chapter 207LLaMA Architecture › Model Architecture

Reading section 3 of 11 · Model Architecture · 15 minSection 3 of 11 · 15 min

Section 3 of 11 · 15 min read

LLaMA Architecture › Model ArchitectureOne section · Sections in this chapter3 / 11 ↓

Sections in this chapter11

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

LLaMA follows the decoder-only transformer architecture, the standard for autoregressive language models. Each input token passes through an embedding layer, a stack of transformer blocks, a final normalization layer, and an output projection that produces vocabulary logits. The decoder-only design means the model predicts the next token given all previous tokens, using causal masking to prevent attention to future positions.

Decoder-Only Architecture

In a decoder-only transformer, each position can only attend to itself and previous positions. This causal structure enables autoregressive generation: the model predicts one token at a time, using its own predictions as input for the next step. The architecture differs from encoder-only models (like BERT, which see all positions) and encoder-decoder models (like T5, which have separate components for input processing and output generation).

The architecture consists of these core components:

In[5]:

Code

# LLaMA model sizes and their configurations
llama_configs = {
    "7B": {
        "d_model": 4096,
        "n_layers": 32,
        "n_heads": 32,
        "d_ff": 11008,  # SwiGLU: ~2.67 * d_model, rounded
        "vocab_size": 32000,
        "context_length": 2048,
    },
    "13B": {
        "d_model": 5120,
        "n_layers": 40,
        "n_heads": 40,
        "d_ff": 13824,
        "vocab_size": 32000,
        "context_length": 2048,
    },
    "33B": {
        "d_model": 6656,
        "n_layers": 60,
        "n_heads": 52,
        "d_ff": 17920,
        "vocab_size": 32000,
        "context_length": 2048,
    },
    "65B": {
        "d_model": 8192,
        "n_layers": 80,
        "n_heads": 64,
        "d_ff": 22016,
        "vocab_size": 32000,
        "context_length": 2048,
    },
}

Out[6]:

Console

LLaMA Model Configurations
======================================================================
Model         Dim   Layers    Heads    FFN Dim    Context
----------------------------------------------------------------------
7B          4,096       32       32     11,008      2,048
13B         5,120       40       40     13,824      2,048
33B         6,656       60       52     17,920      2,048
65B         8,192       80       64     22,016      2,048
----------------------------------------------------------------------
All models use vocabulary size: 32,000 tokens

Several patterns emerge from these configurations. First, the model dimension and number of heads scale together, maintaining a constant head dimension of 128. This head dimension balances expressiveness with computational efficiency. Second, the FFN dimension follows the SwiGLU formula: approximately $\frac{8}{3} \times d_{\text{model}}$ , rounded for hardware efficiency. Third, all models share the same vocabulary and context length, simplifying deployment across model sizes.

Out[7]:

Visualization

Log-log line chart of model dimension versus parameter count for LLaMA 7B, 13B, 33B, and 65B. — Model dimension increases from 4,096 in LLaMA 7B to 8,192 in LLaMA 65B as parameter count grows.

Log-log line chart of feed-forward dimension versus parameter count for the four LLaMA model sizes. — FFN dimension rises from 11,008 to 22,016 while preserving the SwiGLU expansion ratio.

Out[8]:

Visualization

Log-log line chart of transformer layer count versus parameter count for the four LLaMA model sizes. — Transformer depth increases from 32 to 80 layers across the LLaMA model family.

Log-log line chart of attention head count versus parameter count for the four LLaMA model sizes. — Attention head count grows from 32 to 64 while the per-head dimension stays fixed at 128.

The scaling reveals important design constraints. The FFN dimension maintains the SwiGLU ratio ( $\approx 2.67 \times d_{\text{model}}$ ) consistently across sizes. The number of attention heads grows more slowly than model dimension, keeping the per-head dimension constant at 128.

The Transformer BlockLink Copied

Each LLaMA transformer block contains two sublayers: multi-head self-attention and a feed-forward network. What distinguishes LLaMA from the original transformer is the normalization strategy and activation function.

Out[9]:

Visualization

Block diagram showing the flow through a LLaMA transformer block: input to RMSNorm to Multi-Head Attention with RoPE, residual add, then RMSNorm to SwiGLU FFN, residual add, to output. — Architecture of a single LLaMA transformer block. RMSNorm is applied before each sublayer (pre-norm), and residual connections bypass both sublayers. The attention sublayer uses RoPE for position encoding, while the FFN uses SwiGLU activation.

The pre-norm configuration places normalization before each sublayer rather than after. To understand why this matters, consider what happens during training when gradients flow backward through dozens of layers. In the original "post-norm" transformer, normalization happens after adding the residual, which means the residual path passes through a normalization operation before the gradient continues. In deep networks, this can destabilize gradient magnitudes and require careful warm-up schedules to avoid early training collapse. Pre-norm moves normalization inside the residual branch, leaving the skip connection clean. Gradients can flow back through the addition operation with no normalization in the way, which provides a stable highway for gradient information even in networks with 80 or more layers.

Mathematically, a LLaMA block computes:

\begin{aligned} \mathbf{h} &= \mathbf{x} + \text{Attention}(\text{RMSNorm}(\mathbf{x})) \\ \mathbf{y} &= \mathbf{h} + \text{FFN}(\text{RMSNorm}(\mathbf{h})) \end{aligned}

where:

$\mathbf{x} \in \mathbb{R}^{n \times d}$ : input to the block with $n$ tokens and dimension $d$
$\text{RMSNorm}(\cdot)$ : root mean square normalization
$\text{Attention}(\cdot)$ : multi-head self-attention with RoPE
$\text{FFN}(\cdot)$ : SwiGLU feed-forward network
$\mathbf{h}$ : intermediate representation after attention
$\mathbf{y}$ : block output

The residual connections add the sublayer output to the unmodified input, creating direct gradient paths through the network. This is necessary for training stability in deep networks: gradients can flow through the residual paths without encountering the transformations that might attenuate or distort them.

Notice that the block always adds the sublayer output back to the original input, not to some processed version of it. This means each block computes a small "correction" to the current representation rather than fully replacing it. A layer that learned nothing useful could simply output zeros, leaving the input unchanged through the residual path. This graceful fallback makes it easier to train very deep networks, because a bad layer causes minimal damage rather than catastrophic feature destruction.

Key Architectural ChoicesLink Copied

LLaMA's specific component choices each address known issues with earlier transformer designs:

RMSNorm over LayerNorm: RMSNorm normalizes by the root mean square without mean centering. This saves computation (one fewer reduction operation) while empirically matching LayerNorm's quality.

Out[10]:

Visualization

Overlaid histograms of input activations, LayerNorm outputs, and RMSNorm outputs from minus five to five. — Activation distributions before and after LayerNorm and RMSNorm. Both methods produce similarly concentrated outputs despite RMSNorm omitting mean centering.

Box plots of root mean square values for inputs, LayerNorm outputs, and RMSNorm outputs, with a dashed target line at one. — Root mean square distributions before and after normalization. LayerNorm and RMSNorm both concentrate output scale near the target value of one.

Both methods produce outputs with consistent scale (RMS near 1.0), which is the primary goal of normalization for training stability. RMSNorm achieves this with one fewer reduction operation per vector. Skipping the mean computation saves approximately 15-20% of normalization cost.

In practice, the mean of a typical transformer activation vector is already close to zero for most of training. The distribution is approximately symmetric around zero because activations pass through operations that do not systematically introduce bias. If the mean is already near zero, subtracting it changes very little while adding computational cost. RMSNorm observes this empirically and eliminates the step entirely. The result is simpler code, faster execution, and essentially identical behavior during training.

Given an input vector, RMSNorm computes:

\text{RMSNorm}(\mathbf{x}) = \frac{\mathbf{x}}{\sqrt{\frac{1}{d}\sum_{i=1}^{d} x_i^2 + \epsilon}} \odot \boldsymbol{\gamma}

where:

$\mathbf{x} \in \mathbb{R}^d$ : input vector (a single token's representation)
$d$ : dimension of the input vector
$x_i$ : the $i$ -th element of $\mathbf{x}$
$\epsilon$ : small constant (typically $10^{-6}$ ) for numerical stability
$\boldsymbol{\gamma} \in \mathbb{R}^d$ : learnable scale parameters
$\odot$ : element-wise (Hadamard) multiplication

The denominator computes the root mean square of all elements, effectively measuring the "average magnitude" of the vector. Dividing by this value ensures consistent activation scales across layers.

SwiGLU over ReLU or GELU: The feed-forward network uses a gated linear unit with Swish activation. This provides expressive nonlinearity with smooth gradients.

Out[11]:

Visualization

Line plot comparing ReLU, GELU, and Swish activation functions over the range -4 to 4, showing ReLU's hard zero for negative values versus the smooth curves of GELU and Swish. — Comparison of activation functions used in transformers. Swish provides smooth gradients everywhere (unlike ReLU's dead zone for negative inputs) while maintaining nonlinearity. The gating mechanism in SwiGLU multiplies two such activations.

The key difference is visible for negative inputs: ReLU outputs exactly zero (the "dying ReLU" problem), while Swish maintains a small negative output that preserves gradient flow. This smooth gradient behavior helps training stability, especially in deep networks.

The SwiGLU FFN transforms each token representation as:

\text{FFN}(\mathbf{x}) = (\text{Swish}(\mathbf{x}\mathbf{W}_{\text{gate}}) \odot \mathbf{x}\mathbf{W}_{\text{up}}) \mathbf{W}_{\text{down}}

where:

$\mathbf{x} \in \mathbb{R}^d$ : input token representation
$\mathbf{W}_{\text{gate}} \in \mathbb{R}^{d \times d_{\text{ff}}}$ : gate projection matrix
$\mathbf{W}_{\text{up}} \in \mathbb{R}^{d \times d_{\text{ff}}}$ : up projection matrix
$\mathbf{W}_{\text{down}} \in \mathbb{R}^{d_{\text{ff}} \times d}$ : down projection matrix
$d_{\text{ff}}$ : hidden dimension of the FFN (approximately $2.67d$ for SwiGLU)
$\text{Swish}(z) = z \cdot \sigma(z)$ : Swish activation, where $\sigma$ is the sigmoid function
$\odot$ : element-wise multiplication between gate and up projections

The gating mechanism allows the network to selectively process information: the Swish-activated gate path controls how much of each hidden dimension passes through. The trade-off is three weight matrices instead of two, compensated by reducing the hidden dimension from $4d$ to approximately $2.67d$ .

Think of the gate as a learned filter. For each hidden dimension, the gate value is a number between 0 and 1 (soft-gated by the Swish function) that decides how much the corresponding "content" value contributes to the output. If the gate is near zero for a particular dimension, that dimension is suppressed regardless of what the content branch computed. If it is near one, the content passes through fully. This selective suppression allows the FFN to dynamically focus on the most relevant features for a given input. This provides a form of input-dependent computation that plain linear transformations cannot achieve.

The reduction from $4d$ to $2.67d$ deserves explanation. The original transformer FFN used two matrices: one expanding from $d$ to $4d$ , one contracting back. SwiGLU adds a third matrix (the gate projection) but to keep total parameter count comparable, the hidden dimension shrinks. With two standard matrices, you have $2 \times 4d \times d = 8d^2$ parameters. With SwiGLU's three matrices at $2.67d$ , you have $3 \times 2.67d \times d \approx 8d^2$ parameters. The count stays roughly the same, but the model gains expressiveness through the gating structure.

RoPE over learned or sinusoidal positions: Rotary position embedding encodes position through rotation applied to query and key vectors. This naturally produces relative position awareness: the dot product between rotated vectors depends only on the distance between positions, not their absolute locations. RoPE also enables context length extension through interpolation techniques, which became necessary for LLaMA 2 and LLaMA 3 as context windows expanded well beyond the original 2048 tokens.

We examine each of these components in detail in the next chapter.