Pre-Norm vs Post-Norm › The Original Transformer: Post-Norm
Language AI HandbookAll chapters Pre-Norm vs Post-Norm › The Original Transformer: Post-NormOne section · Sections in this chapter2 / 18 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The 2017 "Attention Is All You Need" paper introduced what we now call the post-norm configuration. In this design, each sublayer (attention or feed-forward) follows a specific pattern: apply the sublayer transformation, add the residual, then normalize.
The intuitive appeal of post-norm is worth appreciating before we explain its limitations. The architecture is conceptually clean: each block receives an input, transforms it through its sublayer, combines the transformation with the original signal via the residual, and then normalizes the result back to a well-behaved distribution. The next block therefore always receives input with predictable statistical properties, zero mean and approximately unit variance. This predictability was expected to be a virtue, and for the relatively shallow models of 2017, it largely was.
The original transformer used only six encoder layers and six decoder layers. At that depth, the gradient flow problems associated with post-norm are manageable. The paper's authors also used warmup schedules and carefully tuned learning rates, practices that paper over the stability issues without eliminating them. It was only when researchers attempted to train models with 24, 48, or more layers that the structural fragility of post-norm became impossible to ignore.
The post-norm formulation was not chosen carelessly. In 2017, residual networks in computer vision (ResNets) were already demonstrating the power of skip connections, and the transformer borrowed this idea directly. The LayerNorm placement after the residual addition mirrored some conventions from batch-normalized ResNets. The pre-norm alternative was proposed in a 2018 paper by Chen et al. ("The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation") and was further analyzed theoretically by Xiong et al. in 2020 ("On Layer Normalization in the Transformer Architecture"). By 2020, the field had largely converged on pre-norm for large models, though the debate about whether post-norm achieves marginally better final quality given sufficient training budget continues in research circles today.
In post-norm transformers, layer normalization is applied after the residual addition. For a sublayer function , the output is:
where:
- : the input representation (a -dimensional vector for each position)
- : the sublayer transformation (either self-attention or feed-forward network)
- : the residual connection, adding the original input to the sublayer output
- : layer normalization applied to the combined signal
- : the output of the block, which becomes the input to the next block
The intuition behind post-norm is straightforward. The sublayer computes its transformation, the residual connection preserves the original signal, and layer normalization ensures the combined output has stable statistics. Each component does one job, and they compose cleanly.
Let's implement a post-norm transformer block to see this in action:
import numpy as np
def layer_norm(x, gamma, beta, eps=1e-5):
"""
Apply layer normalization.
Args:
x: Input of shape (seq_len, d_model)
gamma: Scale parameter of shape (d_model,)
beta: Shift parameter of shape (d_model,)
eps: Small constant for numerical stability
Returns:
Normalized output of shape (seq_len, d_model)
"""
mean = x.mean(axis=-1, keepdims=True)
var = x.var(axis=-1, keepdims=True)
x_norm = (x - mean) / np.sqrt(var + eps)
return gamma * x_norm + beta
def post_norm_block(x, sublayer_fn, gamma, beta):
"""
Post-norm block: Sublayer -> Add -> Norm
Args:
x: Input tensor
sublayer_fn: Function computing the sublayer transformation
gamma, beta: LayerNorm parameters
Returns:
Output after post-norm block
"""
# Apply sublayer
sublayer_output = sublayer_fn(x)
# Add residual
residual_sum = x + sublayer_output
# Normalize
output = layer_norm(residual_sum, gamma, beta)
return outputThe ordering is explicit: sublayer first, residual addition second, normalization third. This creates a clean signal flow where the normalized output is always what enters the next layer.
# Demonstrate post-norm with a simple sublayer
np.random.seed(42)
seq_len, d_model = 4, 8
# Input and normalization parameters
x = np.random.randn(seq_len, d_model)
gamma = np.ones(d_model)
beta = np.zeros(d_model)
# Simple sublayer (simulating attention or FFN)
W = np.random.randn(d_model, d_model) * 0.1
def simple_sublayer(x):
return np.tanh(x @ W)
# Apply post-norm block
output_post = post_norm_block(x, simple_sublayer, gamma, beta)Post-Norm Block Statistics ======================================== Input mean: -0.1373, std: 0.9311 Output mean: 0.0000, std: 1.0000
The output has approximately zero mean and unit variance (std 1), confirming that layer normalization has done its job. Regardless of what the sublayer computes or how the residual shifts the distribution, the final normalization step ensures stable statistics. This normalized output is what enters the next block in the network, preventing activation magnitudes from growing unboundedly.