Batch Normalization › The Smoothing Effect on the Optimization Landscape

To appreciate why batch normalization helps so much beyond just normalization, it helps to visualize what it does to the loss landscape. Without batch normalization, the loss surface for a deep network is highly irregular: it has many narrow valleys, sharp cliffs, and regions where the gradient changes direction dramatically over small steps. The same gradient descent step that works well in one region of this landscape may cause divergence in another.

Batch normalization smooths this landscape through two related effects. First, by keeping activations in a controlled range, it prevents the extreme activation magnitudes that create the sharpest features of the loss surface. Second, as the Santurkar et al. analysis showed, it reduces the magnitude of the Hessian of the loss (the matrix of second derivatives), which directly bounds how much the gradient can change between consecutive gradient steps. This bound, called the $\beta$-smoothness of the loss function, determines how large a learning rate can safely be used: specifically, gradient descent is guaranteed to converge for step sizes up to $1/\beta$.

The practical implication is a wider range of usable learning rates. Learning rate tuning for deep networks is notoriously difficult: too small, and training is slow; too large, and training diverges. Batch normalization widens the range of learning rates that produce stable training, making the optimization procedure more forgiving. This is why the original batch normalization paper was able to train an Inception architecture with learning rates 14 times larger than the baseline, achieving the same validation accuracy in 14 times fewer training steps.