Batch Normalization › Running Statistics for Inference

During training, batch normalization uses statistics computed from the current mini-batch: $\mu_{\mathcal{B}}$ and $\sigma^2_{\mathcal{B}}$ change with every batch. This works because mini-batches are reasonably representative of the full dataset, and the stochastic nature of batch statistics even provides a mild regularization effect (discussed further below).

At inference time, however, there is typically only a single sample or a small batch that may not represent the training distribution. Computing statistics over a single sample would be meaningless: the "mean" of a single value is that value itself, and the "variance" is zero, so normalization would map every single input to zero regardless of its content. Instead, batch normalization switches to using fixed statistics estimated from the entire training set.

These fixed estimates, called running statistics or population statistics, are accumulated during training using exponential moving averages:

$$\begin{aligned} \mu_{\text{running}} &\leftarrow (1 - \alpha) \cdot \mu_{\text{running}} + \alpha \cdot \mu_{\mathcal{B}} \\ \sigma^2_{\text{running}} &\leftarrow (1 - \alpha) \cdot \sigma^2_{\text{running}} + \alpha \cdot \sigma^2_{\mathcal{B}} \end{aligned}$$

where:

• $\mu_{\text{running}}$: the exponential moving average of batch means, updated after each training batch
• $\sigma^2_{\text{running}}$: the exponential moving average of batch variances, updated after each training batch
• $\alpha$: the momentum parameter controlling how quickly the running estimates update (typically 0.1 in PyTorch, meaning new batches contribute 10%)

At inference time, normalization uses these stable estimates:

$$\hat{x}_i = \frac{x_i - \mu_{\text{running}}}{\sqrt{\sigma^2_{\text{running}} + \epsilon}}$$

This means batch normalization layers have two distinct behaviors: a training mode that uses batch statistics and an inference mode that uses running statistics. In PyTorch, this is controlled with model.train() and model.eval().

The momentum parameter $\alpha = 0.1$ is a careful engineering choice. A larger $\alpha$ causes the running statistics to update more aggressively toward each new batch, which makes them track recent batches closely but can cause instability if the data distribution is noisy or the batch size is small. A smaller $\alpha$ makes the running estimates more stable but slower to converge to the true population statistics. The PyTorch default of 0.1 is a reasonable balance for most training setups with moderate batch sizes.

One subtlety worth understanding: the running statistics are not trained by gradient descent. They are updated via this exponential moving average rule and are never part of the computational graph during the forward pass in training mode. They are purely a bookkeeping mechanism for inference. This is why they are registered as "buffers" in PyTorch rather than "parameters": they are saved with the model state but not updated by the optimizer.

The figure below illustrates how running statistics converge toward the true dataset statistics over training batches, while individual batch estimates fluctuate around them.