Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Pre-Norm vs Post-Norm › The Pre-Norm Alternative

Michael BrenndoerferUpdated June 10, 20253 min read

Language AI HandbookAll chapters

Part 15 of 67Transformer Blocks

Chapter 106Pre-Norm vs Post-Norm › The Pre-Norm Alternative

Reading section 3 of 18 · The Pre-Norm Alternative · 3 minSection 3 of 18 · 3 min

Section 3 of 18 · 3 min read

Pre-Norm vs Post-Norm › The Pre-Norm AlternativeOne section · Sections in this chapter3 / 18 ↓

Sections in this chapter18

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The pre-norm configuration emerged from research on training very deep networks. Instead of normalizing after the residual, pre-norm normalizes before applying the sublayer.

The key insight behind pre-norm is a reframing of what normalization is for. In post-norm, normalization is a cleanup step: we make a mess (sublayer computation plus residual addition), then tidy it up before handing the result to the next block. In pre-norm, normalization is a preparation step: we tidy up the input before the sublayer does its work. This ensures the sublayer always operates on clean, well-scaled data.

This reframing has a critical consequence for the residual connection. In pre-norm, the residual path carries the raw, unnormalized input directly to the output. Nothing rescales it. Nothing gates it. The signal flows through unmodified. During the forward pass, this means the output will not be normalized, which is the trade-off we quantify below. But during the backward pass, it means gradients have a direct, unimpeded highway back to the earliest layers of the network, and this is what enables stable training at scale.

Pre-Norm Formulation

In pre-norm transformers, layer normalization is applied before the sublayer. For a sublayer function $\text{Sublayer}(\cdot)$ , the output is:

\mathbf{x}' = \mathbf{x} + \text{Sublayer}(\text{LayerNorm}(\mathbf{x}))

where:

$\mathbf{x} \in \mathbb{R}^d$ : the input representation (a $d$ -dimensional vector for each position)
$\text{LayerNorm}(\mathbf{x})$ : layer normalization applied to the input before the sublayer
$\text{Sublayer}(\cdot)$ : the sublayer transformation (either self-attention or feed-forward network), operating on the normalized input
$\mathbf{x} + \cdots$ : the residual connection, adding the original (unnormalized) input to the sublayer output
$\mathbf{x}' \in \mathbb{R}^d$ : the output of the block, which is not normalized

Notice the key difference: normalization happens inside the residual branch, not after the addition. This seemingly minor change alters how gradients flow.

In[6]:

Code

def pre_norm_block(x, sublayer_fn, gamma, beta):
    """
    Pre-norm block: Norm -> Sublayer -> Add

    Args:
        x: Input tensor
        sublayer_fn: Function computing the sublayer transformation
        gamma, beta: LayerNorm parameters

    Returns:
        Output after pre-norm block
    """
    # Normalize first
    x_norm = layer_norm(x, gamma, beta)

    # Apply sublayer to normalized input
    sublayer_output = sublayer_fn(x_norm)

    # Add residual (to original x, not normalized x)
    output = x + sublayer_output

    return output

The ordering change is subtle: normalize, apply sublayer, add residual. The residual connection bypasses both the normalization and the sublayer, creating a direct gradient path from output to input.

In[7]:

Code

# Apply pre-norm block to the same input
output_pre = pre_norm_block(x, simple_sublayer, gamma, beta)

Out[8]:

Console

Pre-Norm Block Statistics
========================================
Input mean:  -0.1373, std: 0.9311
Output mean: -0.1117, std: 0.9280

Notice the key difference: the output statistics differ from the normalized values we saw with post-norm. The output is not normalized because the residual adds the original (unnormalized) input to the sublayer output. This accumulation behavior becomes important when we stack many blocks, as activations can grow in magnitude through the network.