Layer Normalization › In Practice: Using Layer Normalization Effectively

Understanding layer normalization in theory is one thing; applying it correctly in real models requires a few additional considerations that are easy to overlook.

Parameter initialization and scale. The default initialization of $\gamma = 1$ and $\beta = 0$ is appropriate in most cases, but some practitioners have found that initializing $\gamma$ to a smaller value (like $0.1$ or even $0$) at the start of training can help with particularly deep or wide models. The motivation is that smaller $\gamma$ values reduce the signal amplitude in early layers, giving the optimizer more room to maneuver before gradients become large. The T5 paper and some other work explore this idea in the context of large-scale pretraining.

Interaction with weight decay. The $\gamma$ and $\beta$ parameters of layer normalization are typically excluded from weight decay regularization. This is because weight decay penalizes large parameter values, and penalizing $\gamma$ away from zero would interfere with the normalization's ability to recover the original distribution when needed. Most modern deep learning frameworks and training recipes exclude all normalization parameters from weight decay by default, but it's worth verifying this in your own training setup.

Combining with other normalization. Some architectures combine layer normalization with other normalization techniques. For instance, Batch Normalization followed by Layer Normalization is occasionally used in hybrid architectures that process both image patches and text tokens. When combining normalizations, be careful about which dimension each normalization operates on: if two normalizations operate on the same dimension, the second one may undo some of the statistical correction applied by the first.

Memory considerations. Layer normalization requires storing the mean and variance for each token during the forward pass, so they can be used in the backward pass. For a model processing sequences of length $L$ with hidden dimension $d$ and $N$ layers, this adds $O(N \cdot L \cdot 2)$ stored scalars per sample. In practice, this memory cost is negligible compared to the activations themselves, but it is worth keeping in mind when debugging out-of-memory errors in very long sequence models.

Gradient accumulation and normalization. When using gradient accumulation (accumulating gradients over multiple mini-batches before applying an update), layer normalization behaves correctly because it computes statistics independently for each sample. In contrast, batch normalization would see artificially small batches at each forward pass, producing noisier statistics. This is one reason why large language model training almost exclusively uses layer normalization: it remains well-behaved under the gradient accumulation strategies necessary for training on long sequences.