Layer Normalization › Summary

Layer normalization is a standard component of transformer architectures because it stabilizes training in deep models. Unlike batch normalization, which computes statistics across the batch dimension, layer normalization operates on each sample independently. This makes it well-suited for variable-length sequences and small batch sizes.

The core operation normalizes each token's representation to zero mean and unit variance, then applies learned scale ($\gamma$) and shift ($\beta$) parameters to recover representational flexibility. This simple transformation stabilizes activations throughout the network, prevents gradient issues during training, and reduces sensitivity to initialization. The backward pass through layer normalization distributes gradient information across all features through the mean and variance coupling, which helps prevent individual features from dominating the gradient signal.

The placement of layer normalization within transformer blocks has evolved from post-norm (original transformer) to pre-norm (modern language models). Pre-norm preserves a clean gradient highway through the residual stream, enabling training without extensive warmup schedules and making it the default choice for large-scale pretraining. The tradeoff is that the residual stream can grow with depth, a problem addressed in practice by a final layer normalization before the output head.

Key takeaways:

• Feature-wise normalization: Layer normalization computes mean and variance across the feature dimension, treating each token independently
• Learnable parameters: $\gamma$ and $\beta$ allow the network to undo normalization when beneficial, preserving representational capacity
• Placement matters: Pre-norm (normalize before sublayer) has become the modern standard, improving gradient flow in deep networks
• Epsilon for stability: A small constant prevents division by zero with near-constant inputs; larger values ($10^{-4}$ or more) may be needed for FP16 training
• No batch dependency: Works with any batch size, including single samples during inference
• Gradient coupling: The shared mean and variance computation means every feature's gradient depends on all other features, distributing the learning signal across the representation