Weight Initialization › Limitations and Practical Guidance

Weight initialization, while important, is not a complete solution to training difficulties.

Initialization is only the starting point. A good initialization establishes favorable conditions for learning, but it cannot compensate for fundamental problems like inappropriate architecture choices, insufficient data, or training hyperparameters that are far off. A network with He initialization will still fail if the learning rate is too large by several orders of magnitude.

Normalization techniques reduce sensitivity. Batch normalization and layer normalization have made precise initialization less critical for many architectures. Modern transformers like BERT and GPT demonstrate that a simple, fixed initialization standard deviation works well when combined with layer normalization. The normalization layers continuously adjust the activation distribution during training, absorbing initialization errors.

Very deep networks remain challenging. For networks with hundreds of layers, even careful initialization may not prevent early training instabilities. Additional techniques like learning rate warmup, gradient clipping, and careful residual branch scaling become necessary. The next chapter covers gradient clipping in detail.

The derivations for Xavier and He initialization make assumptions that do not always hold. They assume weights and inputs are independent, that inputs have zero mean, and that the activation function behaves as modeled (linear for Xavier, variance-halving for He). In practice, these assumptions are approximately but not exactly satisfied, particularly after a few training steps. The initializations work well in practice despite these approximations because the variance analysis captures the dominant first-order behavior.

As a practical decision guide:

• Use He initialization for ReLU and GELU networks (the current default for feedforward and convolutional architectures). This is nn.init.kaiming_normal_() or nn.init.kaiming_uniform_() in PyTorch.
• Use Xavier initialization for tanh and sigmoid networks. This is nn.init.xavier_normal_() or nn.init.xavier_uniform_() in PyTorch.
• Use orthogonal initialization for RNN hidden-to-hidden matrices, where the same matrix is applied many times.
• Follow the BERT/GPT-2 approach for transformer architectures with layer normalization: use $\mathcal{N}(0, 0.02)$ for all weights with residual scaling applied to projection layers.
• If you are uncertain about which activation a layer feeds, default to He initialization; it is more tolerant of mismatch than Xavier because the $\sqrt{2}$ factor provides a buffer against variance decay.