Batch Normalization › Batch Normalization as Regularization

An interesting side effect of batch normalization is regularization. Because normalization statistics are computed over the current mini-batch rather than the full dataset, they introduce a form of stochastic noise. Each mini-batch produces slightly different mean and variance estimates, which vary the normalization applied to each sample depending on what other samples appear in the same batch.

Consider a concrete example: sample $A$ appears in one batch alongside samples $B, C, D$, and in another batch alongside samples $E, F, G$. In the first batch, $A$'s activation might be slightly above the mean; in the second batch, it might be slightly below. The network sees $A$ normalized differently each time it appears during training. This variation forces the network to learn representations that are robust to small distributional perturbations, which is exactly the property that improves generalization.

This noise acts similarly to dropout: it prevents the network from relying too heavily on any specific activation pattern, improving generalization. In practice, networks trained with batch normalization often require less or even no dropout. The original batch normalization paper reported that the regularization effect of batch normalization was strong enough to remove the need for dropout in their Inception architecture, achieving better results on ImageNet without dropout than the baseline with dropout.

The strength of this regularization effect scales with the "noisiness" of the batch statistics, which in turn scales with the inverse of the batch size. Small batches produce noisier statistics and more regularization. Large batches produce cleaner statistics and less regularization. This creates a subtle tradeoff: increasing batch size improves the stability of batch normalization as a normalization technique, but simultaneously reduces its regularization benefit. Some researchers argue that this is why models trained with very large batch sizes (thousands of samples) sometimes generalize worse than those trained with moderate batch sizes, despite converging faster.

However, this regularization property disappears at inference time, when the stable running statistics are used instead of stochastic batch statistics. This is by design: at inference time, we want deterministic, stable predictions. The regularization effect was only ever intended as a training-time phenomenon.