Layer Normalization › Layer Normalization

Batch normalization transformed how we train deep feedforward networks, but it stumbles when applied to transformers. The batch dimension becomes problematic: batch sizes vary, sequences have different lengths, and the statistics computed across a batch of diverse sentences lack semantic coherence. Layer normalization, introduced by Ba et al. (2016), sidesteps these issues entirely by normalizing across features rather than across the batch. This seemingly simple change made layer normalization the default normalization technique for transformers, from the original "Attention is All You Need" architecture to modern large language models.

In this chapter, we'll explore why layer normalization works so well for transformers, how it differs from batch normalization in both computation and behavior, and the subtle implementation details that affect training stability. We'll also examine how the placement of layer normalization within transformer blocks affects learning dynamics, a design choice that has evolved significantly since the original transformer architecture.

To appreciate why layer normalization matters, you need to first understand what can go wrong without it. Deep neural networks are sensitive creatures. The gradient signal that propagates through dozens or hundreds of layers is amplified and attenuated by countless multiplication operations. When activations at one layer are large, subsequent layers receive outsized gradients that cause unstable weight updates. When activations are small, gradients vanish before they can reach the earliest layers. Without intervention, training can become brittle or slow and may fail catastrophically.

Normalization techniques solve this by actively constraining the statistical distribution of activations at each layer. Rather than hoping that careful initialization and learning rate schedules will keep activations well-behaved throughout training, normalization directly resets the distribution at predefined points in the network. The question is not whether to normalize, but how. Different normalization strategies make different tradeoffs between computational cost, statistical quality, and architectural compatibility. Batch normalization was the answer for convolutional networks. Layer normalization turned out to be the answer for transformers.

The key intuition is that transformers process each token as an independent entity. When the model applies attention, each token gathers context from others, but each token's representation still flows through the layers as a distinct vector. It makes sense, then, to normalize each representation individually, based on its own statistics, rather than blending its statistics with those of other samples in the batch. Layer normalization does exactly this, and the result is a normalization scheme that works consistently regardless of batch size, sequence length, or what other samples happen to be processed at the same time.