Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Weight Initialization › Vanishing and Exploding Activations

Michael BrenndoerferApril 25, 20255 min read

Language AI HandbookAll chapters

Part 10 of 67Neural Network Foundations

Chapter 70Weight Initialization › Vanishing and Exploding Activations

Reading section 4 of 19 · Vanishing and Exploding Activations · 5 minSection 4 of 19 · 5 min

Section 4 of 19 · 5 min read

Weight Initialization › Vanishing and Exploding ActivationsOne section · Sections in this chapter4 / 19 ↓

Sections in this chapter19

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Suppose we fix the symmetry problem by using random initialization. We draw weights from a standard normal distribution $\mathcal{N}(0, 1)$ . At first this seems reasonable, but a deep network reveals a severe problem.

In a layer with $n_{\text{in}}$ inputs, each pre-activation sums $n_{\text{in}}$ weighted inputs. If each weight has variance 1 and each input has variance 1, then the pre-activation has variance $n_{\text{in}}$ . Its standard deviation is $\sqrt{n_{\text{in}}}$ . For a layer with 256 inputs, every layer amplifies the signal by a factor of $\sqrt{256} = 16$ .

After 10 layers, the signal has grown by a factor of $16^{10} \approx 10^{12}$ . This is exploding activations: values become astronomically large, causing numerical overflow or saturating activation functions.

The opposite problem occurs with too-small initialization. If weights are drawn from $\mathcal{N}(0, 0.01^2)$ , each layer reduces the signal by a factor of roughly $0.01 \cdot \sqrt{n_{\text{in}}}$ . For 256 inputs this is $0.01 \times 16 = 0.16$ . After 10 layers, the signal shrinks to $0.16^{10} \approx 10^{-8}$ . This is vanishing activations: the network loses the ability to distinguish inputs, and gradients vanish with the activations.

Both failure modes prevent learning. The goal is to find an initialization scale that keeps activations in a reasonable range throughout the network, regardless of depth.

The gradient side of the picture is equally important and often more immediately damaging to training. When activations vanish, the gradients that flow backward during backpropagation vanish with them. This happens because the gradient of the loss with respect to a weight in an early layer requires multiplying many terms together through the chain rule, and if any activation is near zero, its contribution to the gradient chain is near zero. This is the vanishing gradient problem, and it is what made networks with more than a few layers so notoriously difficult to train before principled initialization and techniques like batch normalization were introduced.

When activations explode, the gradients can explode too, driving weight updates to extreme values and destabilizing the optimization. In practice, activation saturation with sigmoid or tanh is another failure mode: once activations reach the flat regions of these functions, their derivatives are essentially zero, and gradients stop flowing regardless of whether the pre-activations have exploded or not.

In[5]:

Code

import numpy as np

np.random.seed(42)


def forward_pass(x, weights, activation="tanh"):
    """Propagate input through multiple layers and return all activations."""
    activations = [x]
    for W in weights:
        x = x @ W
        if activation == "tanh":
            x = np.tanh(x)
        elif activation == "relu":
            x = np.maximum(0, x)
    activations.append(x)
    return activations


# Simulate a 10-layer network with different initialization scales
n_layers = 10
layer_size = 256
batch_size = 100
scales = [0.01, 0.1, 1.0, 2.0]

Out[6]:

Console

Effect of initialization scale on activations (tanh network):
------------------------------------------------------------
Scale 0.01: Final layer std = 0.000000
Scale 0.10: Final layer std = 0.633119
Scale 1.00: Final layer std = 0.973512
Scale 2.00: Final layer std = 0.988212

The output reveals the two failure modes. Scale 0.01 produces nearly zero activations after just 10 layers. Scale 2.0 saturates tanh immediately, because activations hit the $\pm 1$ saturation zone where gradients are nearly zero. Scales 0.1 and 1.0 are better, but neither is precisely calibrated.

Out[7]:

Visualization

Violin plot showing activation distributions narrowing toward zero across 10 layers. — Activation distributions across 10 layers with small initialization (scale=0.01). Each violin shows the distribution of activations in that layer. Activations progressively collapse toward zero, making gradients vanish and preventing learning in deeper layers.

Violin plot showing bimodal activation distributions concentrated at plus and minus one. — Activation distributions with large initialization (scale=2.0). Activations saturate at the extremes of tanh (near plus or minus 1), where the gradient is nearly zero. This also halts learning, just for a different reason.

The violin plots make the failure modes visible. Small initialization compresses all activations toward zero layer by layer. Large initialization immediately saturates them into two spikes near $\pm 1$ . We need a principled way to find the scale that keeps activations in the useful middle range.