Layer Normalization › Key Parameters

When using nn.LayerNorm in PyTorch, understanding the key parameters helps you configure it correctly for your architecture:

• normalized_shape: The shape of the input over which to normalize. For transformers, this is typically the hidden dimension d_model (e.g., 768, 1024). You can also pass a list like [seq_len, d_model] to normalize over multiple dimensions, though normalizing over features only is the standard choice.

• eps: The epsilon value added to the denominator for numerical stability. Default is 1e-5, which works well for most cases. For mixed-precision (FP16) training, you may need a larger value like 1e-4 to avoid underflow issues when variance is very small.

• elementwise_affine: Whether to include learnable $\gamma$ and $\beta$ parameters. Default is True. Setting to False removes the learnable parameters, reducing model size slightly but limiting the network's ability to learn optimal feature scales. Some research has found that removing the affine transform causes minimal performance degradation in very large models, suggesting that the normalization itself, not the learnable parameters, is primarily responsible for the stability benefits.