Batch Normalization › Batch Size Sensitivity

Batch normalization has a significant and often overlooked weakness: it depends fundamentally on batch size. To estimate meaningful statistics, a batch must contain enough samples to represent the data distribution. When batch sizes are very small, typically fewer than 8 to 16 samples, the batch statistics are too noisy to be reliable.

The variance estimate, in particular, becomes unstable with small batches. Consider estimating the variance of a distribution from only 4 samples. The sample variance is an unbiased estimator, but its own variance (the variance of the estimator itself) is proportional to $\frac{2}{m-1}$ times the square of the population variance. For $m = 4$, this means the estimated variance fluctuates wildly from batch to batch, making the normalization highly inconsistent. Different batches may normalize the same input to very different values, creating an unstable training signal.

This creates practical problems in several important scenarios:

• Memory-constrained training: When using very large models or high-resolution images, GPU memory may only accommodate batches of 2 to 4 samples. For example, training a 3D medical image segmentation network on high-resolution volumes may only allow batch sizes of 1 or 2 due to memory constraints.
• Object detection: Architectures like Faster R-CNN operate on single images at a time during the region proposal phase, making batch normalization impractical.
• Recurrent networks: Variable-length sequences complicate the definition of "a batch" for normalization purposes, since different sequences contribute different numbers of timesteps.
• Online learning: Some applications require learning from single samples, which is completely incompatible with batch statistics.
• Multi-GPU training with gradient accumulation: When gradients are accumulated over many small mini-batches before updating weights, the effective batch size seen by each batch normalization layer is still the small per-step batch size, not the accumulated size.

The sensitivity to batch size was a significant practical obstacle in many domains and directly motivated the development of alternative normalization schemes.