Batch Normalization › Where to Place Batch Normalization

One of the most debated aspects of batch normalization is its placement relative to the activation function. The original paper placed batch normalization before the activation (pre-activation), while many practitioners prefer placing it after the activation (post-activation). A third variant, used in residual networks, places batch normalization inside the residual branch. Each approach has distinct tradeoffs.

This section covers the arguments for each placement and the modern consensus, along with the specific considerations for residual architectures.

The original Ioffe and Szegedy paper placed batch normalization between the linear transformation and the activation:

Linear layer $\to$ Batch Norm $\to$ Activation

The argument: activations enter the normalization step before nonlinearity is applied. For sigmoid and tanh, this ensures the inputs land in the sensitive linear region of the activation, preventing saturation. The normalized distribution, centered at zero with unit variance, places most inputs in the range $[-2, 2]$, which is exactly where sigmoid and tanh have meaningful gradients.

For ReLU, pre-activation normalization has a subtle problem. After normalizing, roughly half the activations will be negative (since the distribution is centered at zero). These negative activations get zeroed by ReLU, meaning that half the neurons are effectively dead after each batch normalization layer. The learned shift $\beta$ can compensate somewhat, shifting the distribution so fewer values fall below zero, but this is a real limitation. Networks with many such layers can suffer from significant dead-neuron problems.

Many modern implementations place batch normalization after the activation:

Linear layer $\to$ Activation $\to$ Batch Norm

The argument: this preserves the nonlinear output of the activation before normalization. For ReLU networks, post-activation normalization operates on non-negative values, which are more informative. The normalized values can then be scaled and shifted by $\gamma$ and $\beta$ to the appropriate range for the next layer. Since ReLU outputs are always non-negative, the distribution being normalized is asymmetric, and batch normalization will shift it to have mean close to zero and unit variance, which is a sensible starting distribution for the next layer.

For residual networks, He et al. (2016) proposed a "pre-activation" ordering that places batch normalization before both the activation and the linear layer within the residual branch:

Batch Norm $\to$ Activation $\to$ Linear layer (inside residual branch)

This ordering has a useful property: the signal through the shortcut connection is entirely un-modified by batch normalization. It passes directly from one residual block's output to the next addition. This means gradients can flow cleanly through the shortcut path without being affected by the normalization's scaling, which is particularly beneficial in very deep networks (hundreds of layers) where even small gradient modifications compound severely across depth. He et al. showed that this ordering improved training for networks deeper than 100 layers.

In practice, neither pre-activation nor post-activation placement is universally superior for all cases. For ReLU networks of standard depth, post-activation placement often works slightly better. For very deep networks with residual connections, pre-activation batch normalization tends to improve gradient flow. The practical guidance is to use post-activation as the default and experiment with pre-activation if training is unstable.