Weight Initialization › Summary

Weight initialization determines whether a neural network can learn effectively from the start. The core principle is variance preservation: weights should be scaled so that activations and gradients maintain reasonable magnitudes as they propagate through layers.

Key takeaways:

• Zero initialization causes symmetry collapse: all neurons in a layer learn identically, wasting representational capacity. Any constant initialization has this problem.
• Random initialization introduces scale sensitivity: too small causes vanishing activations, too large causes explosion or saturation
• LeCun initialization uses variance $1/n_{\text{in}}$ to preserve forward pass variance in linear networks. It is the theoretical foundation that Xavier and He build upon.
• Xavier initialization uses variance $2/(n_{\text{in}} + n_{\text{out}})$, balancing forward and backward variance for tanh and sigmoid activations
• He initialization uses variance $2/n_{\text{in}}$, adding a factor of 2 to compensate for ReLU's variance-halving effect
• Orthogonal initialization preserves exact variance in linear transformations, making it valuable for RNNs where spectral radius matters for long-range gradient flow
• BERT and GPT use a simple $\mathcal{N}(0, 0.02)$ approach, relying on layer normalization and residual scaling to maintain stability
• Normalization layers reduce sensitivity to initialization and explain why modern large models need less precise initialization schemes
• Diagnostics matter: examining activation statistics and gradient norms at the start of training can pinpoint initialization problems before wasting compute on a doomed training run

The next chapter covers gradient clipping, a complementary technique that handles gradient explosions that can occur during training even when initialization is correct, particularly in recurrent networks processing long sequences.