Layer Normalization › Why Batch Normalization Fails for Transformers
Language AI HandbookAll chapters Layer Normalization › Why Batch Normalization Fails for TransformersOne section · Sections in this chapter2 / 16 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Before diving into layer normalization, it's worth understanding exactly why batch normalization doesn't work well for sequence models. The core issue is that batch normalization computes statistics across the batch dimension, assuming that each position in a layer sees similar data across samples.
In a transformer processing sentences of varying lengths, each position in the sequence represents something different. Position 0 might be "The" in one sentence and "Scientists" in another. Position 50 might be a verb in one sentence, a noun in another, and padding in a third. Computing a mean and variance across these semantically unrelated positions produces statistics that don't reflect any meaningful property of the data.
Think of it this way: batch normalization operates like a quality control station on an assembly line. It measures the typical behavior of a particular station across many products passing through it, then adjusts each product to conform to that average. This works well when all products are similar, as in image processing where every image has the same spatial structure. But in language, the "products" (tokens) at each position are wildly different across samples. A batch normalization layer at position 10 has to average over "the" from one sentence, "president" from another, and a padding token from a third. The resulting statistics are meaningless.
There are two additional practical problems. First, batch normalization requires a reasonably large batch size to produce stable statistics. With small batches (which are common when training large models on long sequences), the estimated mean and variance become noisy, degrading training. Second, batch normalization maintains running statistics during training that are used at inference time. These running statistics must match the test-time distribution, which can be tricky to ensure and makes model deployment more complex.
Layer normalization sidesteps all of these problems by computing statistics entirely within each sample, independently of the batch. Each token's representation is normalized using only that token's own features. The statistics are exact, not estimated; they don't depend on batch size; and there are no running statistics to maintain. Training and inference use the same computation.
import torch
# Simulate a batch of sequences with varying content
batch_size = 4
seq_len = 8
hidden_dim = 16
# Different sequences have very different activation patterns
activations = torch.randn(batch_size, seq_len, hidden_dim)
# Exaggerate differences between sequences
activations[0] *= 0.5 # First sequence has small activations
activations[1] *= 3.0 # Second sequence has large activations
activations[2] += 5.0 # Third sequence has positive shift
activations[3] -= 5.0 # Fourth sequence has negative shiftActivation statistics per sequence (averaged across positions and features): Sequence 0: mean= 0.041, std=0.481 Sequence 1: mean= 0.135, std=3.056 Sequence 2: mean= 5.027, std=0.976 Sequence 3: mean= -5.042, std=0.935 Batch statistics at position 0: Mean range: [-1.47, 1.68] Var range: [8.38, 28.55]
The batch statistics are dominated by the extreme sequences, and these statistics change dramatically between positions. Layer normalization avoids this problem entirely by computing statistics within each sample independently, treating each token's representation as a self-contained unit to normalize.
Notice that the statistics at position 0 span a wide range because different sequences have wildly different activation magnitudes. Any normalization based on these cross-sample statistics would apply an inappropriate correction to every token in the batch. Layer normalization avoids this problem by never mixing statistics across samples.