Batch Normalization › Worked Example: Batch Norm By Hand

Let's trace through a single batch normalization step with a concrete mini-batch. Suppose we have a batch of 4 samples with activations for one feature dimension:

$$x_1 = 2.0, \quad x_2 = 4.0, \quad x_3 = 3.0, \quad x_4 = 1.0$$

Step 1: Compute the batch mean.

$$\mu_{\mathcal{B}} = \frac{2.0 + 4.0 + 3.0 + 1.0}{4} = \frac{10.0}{4} = 2.5$$

Step 2: Compute the batch variance.

$$\sigma^2_{\mathcal{B}} = \frac{(2.0-2.5)^2 + (4.0-2.5)^2 + (3.0-2.5)^2 + (1.0-2.5)^2}{4}$$ $$= \frac{0.25 + 2.25 + 0.25 + 2.25}{4} = \frac{5.0}{4} = 1.25$$

Step 3: Normalize. Using $\epsilon = 10^{-5}$:

$$\hat{x}_1 = \frac{2.0 - 2.5}{\sqrt{1.25 + 10^{-5}}} \approx \frac{-0.5}{1.118} \approx -0.447$$ $$\hat{x}_2 \approx 1.342, \quad \hat{x}_3 \approx 0.447, \quad \hat{x}_4 \approx -1.342$$

Notice that the normalized values sum to zero and have unit variance: $(-0.447)^2 + (1.342)^2 + (0.447)^2 + (-1.342)^2 = 0.2 + 1.8 + 0.2 + 1.8 = 4.0$, and $4.0 / 4 = 1.0$. The normalization has precisely achieved its goal.

Step 4: Scale and shift. With initial parameters $\gamma = 1.0$ and $\beta = 0.0$ (the default initialization):

$$y_i = \gamma \hat{x}_i + \beta = 1.0 \cdot \hat{x}_i + 0.0 = \hat{x}_i$$

Initially, the transformation is pure normalization. As training proceeds, $\gamma$ and $\beta$ will shift away from these initial values to match the learned optimal distribution. If the downstream layer works best when inputs are scaled by 2 and shifted by 0.5, gradient descent will move $\gamma \to 2.0$ and $\beta \to 0.5$ over the course of training.

What happens in the next training step? If a different batch is drawn with slightly different values, say mean 2.7 instead of 2.5, the normalized values will be different. Sample $x_1 = 2.0$ will be normalized to $\frac{2.0 - 2.7}{\sqrt{\sigma^2}} \approx -0.625$ instead of $-0.447$. This is the stochastic noise of batch normalization. Over many batches, the network learns representations that are robust to this variability.

The figure below shows the effect of batch normalization on activation distributions.