Batch Normalization › Batch Normalization in CNNs vs. NLP

The mechanics of batch normalization differ between convolutional neural networks and sequence models, because these architectures have fundamentally different data shapes and different notions of what constitutes a "feature."

In a CNN, the output of a convolutional layer for a batch of $N$ images has shape $(N, C, H, W)$ where $C$ is the number of channels, $H$ the height, and $W$ the width of the feature map. Batch normalization normalizes each channel independently across the batch and spatial dimensions:

$$\mu_c = \frac{1}{N \cdot H \cdot W} \sum_{n=1}^{N} \sum_{h=1}^{H} \sum_{w=1}^{W} x_{n,c,h,w}$$

This means every spatial location in the same channel of the same image contributes to the mean and variance. The intuition is that a channel in a CNN represents a specific learned feature detector, such as a particular edge orientation or texture, and all spatial locations where that feature appears should be normalized together. Whether the feature appears in the upper left or lower right of the image, it represents the same underlying pattern and should be treated consistently.

Each channel has its own $\gamma_c$ and $\beta_c$ parameters. For a convolutional layer with 256 output channels, batch normalization adds 512 learnable parameters ($256 \times 2$), which is negligible compared to the weight matrix itself but still affects training dynamics.

For CNNs applied to image classification, batch normalization has been essential to achieving state-of-the-art results. ResNet, VGG, Inception, DenseNet, and virtually every other high-performing CNN architecture from 2015 onward uses batch normalization after its convolutional layers. The ability to train networks with dozens to hundreds of layers, which was impractical before batch normalization, directly enabled the advances in image recognition that followed.

For sequence models, the data has shape $(N, T, H)$ where $T$ is the sequence length and $H$ is the hidden dimension. Batch normalization across the batch and time dimensions would mix different timesteps and different sentences, which is conceptually problematic. The statistics of the token at position 1 in one sentence may be very different from the token at position 1 in another sentence (one might be a subject noun, the other an article or punctuation). Normalizing these together would conflate structurally different representations.

Sequences in a mini-batch often have different lengths. The padding tokens used to make sequences the same length would need to be excluded from the statistics computation, which complicates the implementation. And at inference time, a single sequence should produce deterministic outputs, but batch normalization in training mode would give different results depending on what other sequences happen to be in the same batch.

These issues collectively make batch normalization unsuitable for most NLP applications. Layer normalization, which normalizes across the feature dimension $H$ for each individual token position independently, is the standard choice. When you look at transformer code, each multi-head attention block and feedforward layer is followed by layer normalization, not batch normalization. This is not an accident but a deliberate design decision rooted in the differences between image and sequence data.