Layer Normalization (2016) › Limitations

While layer normalization solved many problems with batch normalization, it also introduced its own limitations. The most significant is that layer normalization doesn't provide the same regularization effect that batch normalization does through its dependence on batch statistics. Batch normalization's use of batch statistics introduces noise that acts as a form of regularization, helping models generalize. Layer normalization, being deterministic for a given input, lacks this implicit regularization effect.

This difference means that layer normalization may require stronger explicit regularization, such as dropout or weight decay, to achieve similar generalization performance. In some cases, researchers have found that the combination of layer normalization and appropriate regularization techniques can match or exceed batch normalization's performance, but the regularization must be tuned more carefully.

Another limitation is that layer normalization computes statistics across all features of a single example, which may not always be appropriate. In some architectures, particularly those with very wide layers or with features that have fundamentally different scales or meanings, normalizing across all features might not be optimal. Some variations of layer normalization have been proposed to address this, such as normalizing only across a subset of dimensions or applying different normalization strategies to different parts of the input.

Layer normalization also doesn't address the external covariate shift that occurs when the input distribution changes. Like batch normalization, it normalizes internal activations, but if the input distribution shifts significantly between training and deployment, both techniques can struggle. The learnable scale and shift parameters provide some adaptability, but dramatic distribution shifts may still require retraining or fine-tuning.

For convolutional networks, batch normalization generally remains the preferred choice. The spatial structure of images and the effectiveness of batch statistics in convolutional settings mean that batch normalization typically outperforms layer normalization for vision tasks. While some vision transformers use layer normalization successfully, the technique hasn't replaced batch normalization in standard convolutional architectures.

The computational overhead of layer normalization is slightly different from batch normalization. Layer normalization computes statistics for each example independently, which can be more memory-efficient in some scenarios but requires computing mean and variance for every example rather than once per batch. In practice, the difference is usually negligible, but in very high-throughput inference scenarios, the per-example computation can become a bottleneck.