Batch Normalization › Batch Normalization

Training deep neural networks is notoriously difficult. As networks grow deeper, a subtle problem emerges: the distribution of inputs to each layer shifts continuously during training as the weights in earlier layers update. This forces every downstream layer to constantly adapt to new input statistics, slowing convergence and destabilizing learning. Batch normalization, introduced by Sergey Ioffe and Christian Szegedy in 2015, addresses this directly by normalizing layer inputs using statistics computed from the current mini-batch. The result is one of the most impactful regularization and stability techniques in deep learning, and understanding it deeply is essential for anyone working with modern neural architectures.

In the previous chapter on weight initialization, we saw how proper initialization of weights, through Xavier or He initialization, helps gradients flow cleanly at the start of training. Batch normalization takes this idea further: it actively maintains healthy activation statistics throughout training, not just at initialization. Together, these two techniques form the foundation for training reliably deep architectures. Weight initialization sets the stage; batch normalization keeps it stable as training unfolds.

This chapter covers the full story of batch normalization: why internal covariate shift causes training problems, how the normalization algorithm works step by step, what the running statistics mechanism does at inference time, how batch normalization affects gradient flow and the optimization landscape, where to place it in a network, its limitations and the alternative normalization schemes those limitations inspired, and finally a complete code implementation that makes all of these ideas concrete.