Weight Initialization › Xavier/Glorot Initialization

Our variance analysis derived the optimal weight variance for forward propagation. But training a neural network involves both forward and backward propagation, and gradients face the same variance amplification problem in reverse.

Xavier Glorot and Yoshua Bengio, in their 2010 paper "Understanding the Difficulty of Training Deep Feedforward Neural Networks," recognized that we must consider both signal directions simultaneously. If we optimize only for forward variance, gradients might explode or vanish during backpropagation. The key insight was that a good initialization must care about the health of the gradient signal just as much as the health of the forward activations, because training depends on both flowing cleanly through the entire network.

The forward pass constraint from our derivation above gives:

$$\text{Var}(w) = \frac{1}{n_{\text{in}}}$$

where $n_{\text{in}}$ is the fan-in (number of input connections to each neuron).

During backpropagation, gradients flow in the opposite direction. Each input neuron receives gradient contributions from all $n_{\text{out}}$ output neurons. By the same reasoning applied to the gradient equations, maintaining gradient variance during the backward pass requires:

$$\text{Var}(w) = \frac{1}{n_{\text{out}}}$$

where $n_{\text{out}}$ is the fan-out (number of output connections from each input neuron).

To see why the backward pass requirement involves $n_{\text{out}}$, recall the chain rule during backpropagation. The gradient of the loss with respect to an input activation $x_i$ is:

$$\frac{\partial L}{\partial x_i} = \sum_{j=1}^{n_{\text{out}}} w_{ij} \cdot \frac{\partial L}{\partial z_j}$$

This is a weighted sum of $n_{\text{out}}$ gradient terms, structurally identical to the forward pass equation but with fan-out replacing fan-in. The same variance analysis applies, giving the backward requirement $\text{Var}(w) = 1/n_{\text{out}}$.

We face a dilemma. The forward pass requires $\text{Var}(w) = 1/n_{\text{in}}$, but the backward pass requires $\text{Var}(w) = 1/n_{\text{out}}$. These two constraints conflict whenever $n_{\text{in}} \neq n_{\text{out}}$, which is the common case for layers that change dimensionality.

Glorot and Bengio proposed taking the harmonic mean of the two fan values:

$$\text{Var}(w) = \frac{2}{n_{\text{in}} + n_{\text{out}}}$$

where:

• $n_{\text{in}}$: fan-in (input dimension of the weight matrix)
• $n_{\text{out}}$: fan-out (output dimension of the weight matrix)
• $\frac{2}{n_{\text{in}} + n_{\text{out}}}$: a compromise variance that partially satisfies both the forward and backward variance preservation constraints

Notice that when $n_{\text{in}} = n_{\text{out}}$, this formula reduces to $1/n_{\text{in}}$, recovering the LeCun result. For layers where the input and output dimensions differ substantially, the formula takes a middle path between the two competing requirements. This is an elegant engineering compromise: it is not optimal for either direction alone, but it avoids catastrophically failing in either direction.

This is Xavier initialization (also called Glorot initialization). To sample weights, we convert this target variance to the parameters of either a normal or uniform distribution.

For a normal distribution $\mathcal{N}(0, \sigma^2)$, the variance is $\sigma^2$. Setting $\sigma^2 = 2/(n_{\text{in}} + n_{\text{out}})$ and taking the square root gives the standard deviation:

$$\sigma = \sqrt{\frac{2}{n_{\text{in}} + n_{\text{out}}}}$$

Weights are drawn from:

$$w \sim \mathcal{N}\!\left(0, \sqrt{\frac{2}{n_{\text{in}} + n_{\text{out}}}}\right)$$

For a uniform distribution $U[-a, a]$, the variance is $a^2/3$. Setting $a^2/3 = 2/(n_{\text{in}} + n_{\text{out}})$ and solving for $a$:

$$a = \sqrt{\frac{6}{n_{\text{in}} + n_{\text{out}}}}$$

Weights are drawn from:

$$w \sim U\!\left[-\sqrt{\frac{6}{n_{\text{in}} + n_{\text{out}}}}, \sqrt{\frac{6}{n_{\text{in}} + n_{\text{out}}}}\right]$$

In practice, the uniform variant is the more commonly referenced form, particularly in the original Glorot and Bengio paper. Both versions achieve similar results in practice because the variance formula is the same; only the shape of the distribution differs. The uniform variant is bounded, so it avoids the (very small) probability of drawing a very large outlier weight from the tails of a normal distribution.

The derivation assumes the activation function does not distort the variance of the signal. Tanh and sigmoid satisfy this assumption near zero, where they are both approximately linear. Specifically:

• $\tanh(z) \approx z$ for small $|z|$
• $\sigma(z) \approx 0.25 z + 0.5$ for small $|z|$, with derivative $\approx 0.25$

For tanh, the derivative near zero is approximately 1, meaning the function barely affects variance. For sigmoid, the derivative near zero is about 0.25, which does systematically reduce variance. This is why PyTorch's implementation includes an empirical correction factor of $5/3$ for tanh (accounting for the actual saturation properties) and why some implementations of Xavier for sigmoid include additional adjustment. The original Glorot and Bengio paper noted that sigmoid is more problematic than tanh, because its non-zero mean output shifts the input distribution seen by subsequent layers.