Weight Initialization › Impact on Training Stability

Proper initialization has measurable effects on training stability beyond just convergence speed. The connection is to how well-conditioned the early gradient signal is: well-initialized networks start in regions of the loss surface where gradients are informative and directions are well-scaled, enabling optimizers to take productive steps from the first iteration.

Poor initialization forces the optimizer to spend many epochs correcting the initial state before it can begin learning the actual task. In the worst cases, networks with bad initialization never converge at all, because early gradient signals are too weak (vanishing) or too noisy (exploding) to provide useful update directions. The optimizer takes large steps in meaningless directions, or takes steps so small that any progress is undetectable.

The condition number of the weight matrices is another lens through which to understand initialization quality. When weights are initialized appropriately, the early layers of the network apply transformations with condition numbers close to 1, meaning all directions are scaled similarly. This makes the optimization landscape well-conditioned and gradient descent more efficient. Poor initialization can produce extremely ill-conditioned transformations where some directions are amplified by orders of magnitude more than others, making optimization very difficult.

The interactions between initialization and other training components are also important. Batch normalization dramatically reduces sensitivity to initialization by normalizing activations at each layer during training. Layer normalization, used in transformers, plays a similar role. This is why BERT and GPT can use a simple fixed standard deviation rather than layer-size-dependent formulas: the normalization layers continuously correct the activation distribution during training, absorbing initialization errors.

When normalization layers are absent (some architectures avoid them for specific reasons), initialization quality becomes more critical. In these settings, using the theoretically derived initialization for your specific activation function is the safest strategy.