Batch Normalization › Alternatives to Batch Normalization

The batch-size dependency of batch normalization motivated a family of alternative normalization techniques that compute statistics differently. Understanding these alternatives clarifies what batch normalization does and when to prefer each variant.

This section covers four main alternatives: layer normalization, group normalization, instance normalization, and weight normalization, along with their respective domains of application.

Layer normalization (Ba et al., 2016) normalizes across the feature dimension for each sample independently, rather than across the batch dimension:

$$\mu^{(i)} = \frac{1}{H} \sum_{j=1}^{H} x_{i,j}, \quad \sigma^{2(i)} = \frac{1}{H} \sum_{j=1}^{H} (x_{i,j} - \mu^{(i)})^2$$

where $H$ is the number of features in the layer. Each sample's activations are normalized using statistics computed only from that sample's feature vector, making it completely independent of batch size.

Layer normalization can process a single sample at inference time without any special casing, because the statistics are computed from that single sample's feature vector. There is no distinction between training mode and inference mode: the normalization always uses the current input's own statistics. This simplicity is an advantage over batch normalization.

Layer normalization is particularly well-suited for recurrent and transformer architectures, where sequences vary in length and batch composition is irregular. The original Transformer paper by Vaswani et al. (2017) uses layer normalization throughout, and it has become the default normalization for language models. When you encounter BERT, GPT, or any large language model, layer normalization is doing the heavy lifting of maintaining stable activations. We will encounter layer normalization extensively in the chapters on self-attention and transformer architectures.

The one place where layer normalization underperforms batch normalization is large-batch image classification with CNNs. In this setting, the feature statistics vary substantially across the spatial dimensions (edges look different from backgrounds, for instance), and normalizing across all features of a single spatial position loses useful spatial structure. Batch normalization's approach of normalizing each channel across the batch preserves spatial structure in a way that layer normalization does not.

Instance normalization normalizes each sample and each channel independently, using statistics from that channel's spatial dimensions. For a feature map of height $H$ and width $W$ for sample $i$ and channel $c$:

$$\mu_{i,c} = \frac{1}{HW} \sum_{h=1}^{H} \sum_{w=1}^{W} x_{i,c,h,w}$$

This is the most "local" of the normalization schemes: statistics are computed within each image and within each feature channel separately. No information is shared between images or between channels.

This locality is precisely what makes instance normalization useful for style transfer tasks, where the goal is to separate an image's "content" from its "style." The style of an image is encoded in the channel statistics (means and variances of feature activations), and instance normalization explicitly removes these statistics, effectively stripping away the style. A style transfer network can then re-apply different style statistics through the learned $\gamma$ and $\beta$ parameters, producing images with the content of one image rendered in the style of another.

Group normalization (Wu and He, 2018) divides the feature channels into $G$ groups and normalizes within each group for each sample independently:

$$\mu_{i,g} = \frac{1}{(C/G) \cdot S} \sum_{\text{features in group } g} x_{i,j}$$

where $C$ is the number of channels and $S$ is the number of spatial elements. Group normalization is a generalization that interpolates between layer normalization and instance normalization: when $G = 1$, every channel is in the same group and it becomes layer normalization; when $G = C$, each channel is its own group and it becomes instance normalization.

Group normalization was designed specifically for computer vision tasks where batch sizes must be small due to high-resolution inputs. Wu and He showed that on ImageNet classification with a batch size of 2, group normalization significantly outperforms batch normalization (which degrades severely at this batch size) and is only slightly behind batch normalization at the standard batch size of 32. The key insight is that within each group of channels, there are enough activations to compute reliable statistics even with a single sample, because each channel contributes many spatial positions.

A common choice in practice is $G = 32$ for networks with 256 or more channels, or $G = 8$ for networks with 64 channels. The group size should be chosen so that each group contains a reasonable number of channels (at least 8 to 16) to enable stable statistics.

Weight normalization (Salimans and Kingma, 2016) takes a different approach entirely: instead of normalizing the activations, it normalizes the weight vectors. Each weight vector $\mathbf{w}$ is reparameterized as:

$$\mathbf{w} = \frac{g}{\|\mathbf{v}\|} \mathbf{v}$$

where $\mathbf{v}$ is an un-normalized weight vector and $g$ is a scalar magnitude parameter. This decouples the direction of each weight vector from its magnitude, which stabilizes the gradient updates.

Weight normalization has different properties from activation normalization. It introduces no dependency between training samples and has no distinction between training and inference modes, making it suitable for online learning and recurrent networks where mini-batch dependencies are problematic. It is also computationally cheaper, as it does not require computing statistics over a batch. However, it does not smooth the loss landscape as aggressively as batch normalization, and it requires careful data-dependent initialization to work well.

The normalization methods differ in which dimensions they compute statistics over:

• Batch normalization: statistics over the batch dimension (for each feature independently)
• Layer normalization: statistics over the feature dimension (for each sample independently)
• Instance normalization: statistics over spatial dimensions (for each sample and channel independently)
• Group normalization: statistics over spatial and channel-group dimensions (for each sample and group independently)
• Weight normalization: statistics over the weight vector dimension (no activation statistics involved)

The choice depends on the architecture and task. For large-batch image classification with CNNs, batch normalization typically performs best. For NLP and sequence models with variable-length inputs, layer normalization is standard. For image generation and style transfer, instance normalization is common. For memory-constrained vision tasks, group normalization is the practical choice. For reinforcement learning and online learning scenarios, weight normalization is often preferred.