Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Weight Initialization › Variance Analysis of Forward Propagation

Michael BrenndoerferApril 25, 20255 min read

Language AI HandbookAll chapters

Part 10 of 67Neural Network Foundations

Chapter 70Weight Initialization › Variance Analysis of Forward Propagation

Reading section 5 of 19 · Variance Analysis of Forward Propagation · 5 minSection 5 of 19 · 5 min

Section 5 of 19 · 5 min read

Weight Initialization › Variance Analysis of Forward PropagationOne section · Sections in this chapter5 / 19 ↓

Sections in this chapter19

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The experiments reveal the symptoms, but to find the cure we need to understand the cause mathematically. The key question is: how does the variance of activations change as they pass through each layer?

If we can answer this question, we can work backward to determine what weight variance preserves activation variance across layers. This is the central insight behind principled initialization: treat variance preservation as a design constraint, then solve for the weight scale that satisfies it. The beauty of this approach is that it converts a vague intuition ("weights shouldn't be too big or too small") into a concrete formula with a clear derivation.

The Pre-Activation EquationLink Copied

Consider a single layer with $n_{\text{in}}$ input neurons. Each output neuron computes a weighted sum of its inputs before applying the activation function. For a single output neuron, this pre-activation value is:

z = \sum_{i=1}^{n_{\text{in}}} w_i x_i

where:

$z$ : the pre-activation value (the raw weighted sum before any activation function is applied)
$w_i$ : the weight connecting input neuron $i$ to this output neuron
$x_i$ : the activation value from input neuron $i$
$n_{\text{in}}$ : the number of input neurons (the fan-in)

The pre-activation $z$ is a sum of $n_{\text{in}}$ random terms. We want to understand how its variance relates to the variances of the weights and inputs.

Deriving the Variance RelationshipLink Copied

To derive how variance propagates, we make two assumptions that hold approximately during the early stages of training: weights $w_i$ and inputs $x_i$ are independent random variables with zero mean.

Step 1: Variance is additive for independent random variables. Because the $w_i x_i$ terms are independent of each other (each uses a different weight-input pair):

\text{Var}(z) = \text{Var}\!\left(\sum_{i=1}^{n_{\text{in}}} w_i x_i\right) = \sum_{i=1}^{n_{\text{in}}} \text{Var}(w_i x_i)

Step 2: Variance of a product of two independent zero-mean variables. For any two independent, zero-mean random variables $A$ and $B$ :

\text{Var}(AB) = \mathbb{E}[(AB)^2] - (\mathbb{E}[AB])^2 = \mathbb{E}[A^2]\mathbb{E}[B^2] - 0 = \text{Var}(A) \cdot \text{Var}(B)

Step 3: Combine. Assuming all weights share variance $\text{Var}(w)$ and all inputs share variance $\text{Var}(x)$ :

\text{Var}(z) = \sum_{i=1}^{n_{\text{in}}} \text{Var}(w_i) \cdot \text{Var}(x_i) = n_{\text{in}} \cdot \text{Var}(w) \cdot \text{Var}(x)

where:

$n_{\text{in}}$ : the number of input neurons (fan-in)
$\text{Var}(w)$ : the variance of each weight, assumed identical across all weights in the layer
$\text{Var}(x)$ : the variance of each input activation, assumed identical across all inputs

This formula immediately explains the explosion problem. With $n_{\text{in}} = 256$ and unit-variance weights ( $\text{Var}(w) = 1$ ), the output variance is $256 \times \text{Var}(x)$ . Each layer multiplies variance by $n_{\text{in}}$ , producing exponential growth.

Fan-in and Fan-out

For a weight matrix connecting two layers, fan-in ( $n_{\text{in}}$ ) is the number of input connections per output neuron, and fan-out ( $n_{\text{out}}$ ) is the number of output connections per input neuron. For a fully connected layer with an $n_{\text{in}} \times n_{\text{out}}$ weight matrix, fan-in equals the input dimension and fan-out equals the output dimension.

The assumptions we made (independence, zero mean) are strictly true only at initialization, before any training has occurred. As training proceeds, weights and activations become correlated. However, the variance relationships remain approximately valid throughout early training, which is precisely the period when initialization matters most. By the time the network has trained for many epochs, the optimization landscape has been significantly reshaped and initialization conditions matter much less.

Solving for the Optimal Weight VarianceLink Copied

To maintain stable variance across layers, we want $\text{Var}(z) = \text{Var}(x)$ . Starting from:

\text{Var}(z) = n_{\text{in}} \cdot \text{Var}(w) \cdot \text{Var}(x)

Setting $\text{Var}(z) = \text{Var}(x)$ and dividing both sides by $\text{Var}(x)$ :

1 = n_{\text{in}} \cdot \text{Var}(w)

Solving for the weight variance:

\text{Var}(w) = \frac{1}{n_{\text{in}}}

The weight variance should scale inversely with the number of input connections. Summing $n_{\text{in}}$ independent random terms amplifies variance by $n_{\text{in}}$ , so each weight's variance must be $1/n_{\text{in}}$ to counteract this amplification.

This result is sometimes called LeCun initialization, after Yann LeCun who described it in his 1998 work on convolutional networks. It is the optimal forward-pass initialization for networks with linear or nearly-linear activation functions, and it forms the theoretical foundation for both Xavier and He initialization.

In[8]:

Code

def analyze_variance_propagation(n_layers, layer_size, weight_variance):
    """Track variance across layers with specified weight variance (no activation)."""
    weights = [
        np.random.randn(layer_size, layer_size) * np.sqrt(weight_variance)
        for _ in range(n_layers)
    ]
    x = np.random.randn(1000, layer_size)
    x = x / np.std(x)  # Normalize to unit variance
    variances = [np.var(x)]
    for W in weights:
        x = x @ W
        variances.append(np.var(x))
    return variances


variance_choices = {
    "Too small (1/n^2)": 1 / (layer_size**2),
    "Just right (1/n)": 1 / layer_size,
    "Too large (1)": 1.0,
}

Out[9]:

Console

Variance propagation through 10 layers (no activation):
Layer size: 256
------------------------------------------------------------

Too small (1/n^2):
  Initial variance: 1.0000
  Final variance:   8.2928e-25
  Ratio (final/initial): 8.2928e-25

Just right (1/n):
  Initial variance: 1.0000
  Final variance:   1.0596e+00
  Ratio (final/initial): 1.0596e+00

Too large (1):
  Initial variance: 1.0000
  Final variance:   1.1612e+24
  Ratio (final/initial): 1.1612e+24

With weight variance $1/n$ , the output variance stays close to the input variance, confirming our derivation. The other choices cause variance to explode or collapse exponentially with depth.

Out[10]:

Visualization

Line plot showing variance across 10 layers for three initialization strategies on a log scale, with 1/n remaining stable. — Variance propagation through 10 layers with three different weight initialization scales, measured without activation functions to isolate the linear component. Only the 1/n variance (green) maintains stable variance across all layers. The too-small choice causes exponential decay and the too-large choice causes exponential growth, both visible on the log-scale y-axis.