Weight Initialization › Initialization and Normalization: A Symbiotic Relationship

One of the most important developments in deep learning was the realization that initialization and normalization are complementary tools addressing the same underlying problem: keeping activation distributions healthy throughout a deep network. Understanding their relationship clarifies when each is necessary and when one can substitute for the other.

Batch normalization, introduced by Ioffe and Szegedy in 2015, directly normalizes pre-activation values to have zero mean and unit variance at each layer, then scales and shifts them by learned parameters. This operation continuously corrects the activation distribution during training, reducing the network.s sensitivity to poor initialization. As a result, networks with batch normalization can converge from a much wider range of initializations than those without.

Layer normalization, the variant used in transformers, operates along the feature dimension rather than the batch dimension. It has the same effect of continuously normalizing activations, but it does so per-sample rather than per-batch. This makes it better suited for variable-length sequences and small batch sizes.

The existence of these normalization layers explains a fact that puzzles many practitioners: why does BERT work so well with a simple fixed initialization that theory suggests is suboptimal? The answer is that layer normalization renders the precise initialization scale much less important. Whatever the initial activation distribution, the first layer normalization immediately rescales it. The initialization just needs to avoid catastrophic failure modes (complete collapse or extreme saturation) in the handful of forward passes before the first gradient update.

Without normalization, initialization becomes much more critical. If you design a network that omits batch or layer normalization for any reason, the theoretically derived initializations (Xavier or He, depending on your activation function) are essential for achieving reliable training. This was the regime that the Glorot/Bengio and He et al. papers were addressing, before normalization was ubiquitous.

The practical implication is: if you add batch normalization or layer normalization to your network, you have more flexibility in initialization; if you remove normalization layers for efficiency or other architectural reasons, you should apply principled initialization more carefully.