Batch Normalization › Summary

Batch normalization normalizes layer activations using mini-batch statistics during training, addressing internal covariate shift and dramatically improving the trainability of deep networks. The algorithm involves four steps: computing batch mean, computing batch variance, normalizing to zero mean and unit variance, then applying learned scale $\gamma$ and shift $\beta$ parameters.

Key ideas to retain:

• Internal covariate shift: Without normalization, each layer faces a constantly shifting input distribution as upstream weights update, requiring small learning rates and careful tuning. Batch normalization fixes this by explicitly normalizing layer inputs every forward pass.

• Training vs inference: During training, batch normalization uses batch statistics that change with every mini-batch; during inference, it uses running statistics accumulated via exponential moving averages. Always switch to inference mode before generating predictions.

• Learnable parameters: The $\gamma$ and $\beta$ parameters allow the network to learn the optimal output scale and shift, preserving representational flexibility after normalization. This is what prevents normalization from being a straightjacket on the network's expressiveness.

• Placement: Batch normalization is typically placed after the linear transformation and before or after the activation. For ReLU networks, post-activation placement often works slightly better. For very deep residual networks, pre-activation placement can improve gradient flow.

• Batch size sensitivity: Batch normalization requires reasonably large batches (at least 16, ideally 32 or more) to produce reliable statistics. Small batches cause training instability and poor convergence.

• Smooth loss landscape: Beyond normalization, batch normalization smooths the loss surface and reduces its Lipschitz constant, enabling larger, safer gradient steps. This is likely its primary mechanism of benefit.

• Regularization effect: The stochasticity of batch statistics provides implicit regularization during training, reducing the need for explicit regularization techniques like dropout.

• Alternatives: Layer normalization (default for NLP and transformers), group normalization (small-batch vision tasks), instance normalization (style transfer), and weight normalization each address batch normalization's limitations in specific domains.

In the next chapter, we explore dropout, another foundational regularization technique. Whereas batch normalization addresses training stability through normalization, dropout addresses overfitting by randomly deactivating neurons during training. The two techniques are often used together in modern architectures, and understanding their complementary roles helps clarify why each component is present in the networks you encounter.