Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Vanishing Gradients › Visualizing Gradient Flow Through Time

Michael BrenndoerferMay 8, 20253 min read

Language AI HandbookAll chapters

Part 11 of 67Recurrent Neural Networks

Chapter 76Vanishing Gradients › Visualizing Gradient Flow Through Time

Reading section 8 of 18 · Visualizing Gradient Flow Through Time · 3 minSection 8 of 18 · 3 min

Section 8 of 18 · 3 min read

Vanishing Gradients › Visualizing Gradient Flow Through TimeOne section · Sections in this chapter8 / 18 ↓

Sections in this chapter18

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The easiest way to see vanishing gradients empirically is to train an RNN and track the gradient norm at each timestep. In a healthy training scenario, gradients from all timesteps should have comparable magnitudes. In practice, you see exponential decay.

In[3]:

Code

import numpy as np

np.random.seed(42)


def tanh_deriv(z):
    return 1.0 - np.tanh(z) ** 2


def sigmoid_deriv(z):
    s = 1.0 / (1.0 + np.exp(-z))
    return s * (1.0 - s)


def compute_gradient_norms(
    T, hidden_size, W_spectral_radius, activation="tanh", seed=0
):
    """
    Simulate gradient norms flowing backward through T timesteps.
    Returns a list of gradient norms at each timestep index (from T down to 0).
    """
    rng = np.random.default_rng(seed)
    W = rng.standard_normal((hidden_size, hidden_size))
    eigvals = np.linalg.eigvals(W)
    current_radius = np.max(np.abs(eigvals))
    W = W * (W_spectral_radius / current_radius)

    h = rng.standard_normal(hidden_size) * 0.1
    z_values = []
    for t in range(T):
        z = W @ h + rng.standard_normal(hidden_size) * 0.1
        z_values.append(z)
        if activation == "tanh":
            h = np.tanh(z)
        else:
            h = 1.0 / (1.0 + np.exp(-z))

    grad = np.eye(hidden_size)
    norms = [np.linalg.norm(grad, "fro")]

    for t in reversed(range(T)):
        z = z_values[t]
        if activation == "tanh":
            d = tanh_deriv(z)
        else:
            d = sigmoid_deriv(z)
        J = np.diag(d) @ W
        grad = J @ grad
        norms.append(np.linalg.norm(grad, "fro"))

    return norms

In[4]:

Code

T = 60
hidden_size = 32

scenarios = [
    ("tanh, rho=0.95", "tanh", 0.95, "#1f77b4"),
    ("tanh, rho=0.80", "tanh", 0.80, "#ff7f0e"),
    ("sigmoid, rho=0.95", "sigmoid", 0.95, "#2ca02c"),
    ("sigmoid, rho=0.80", "sigmoid", 0.80, "#d62728"),
]

results = {}
for label, act, rho, color in scenarios:
    norms = compute_gradient_norms(T, hidden_size, rho, activation=act, seed=42)
    results[label] = (norms, color)

Out[5]:

Visualization

Line plot of gradient norms vs timesteps showing exponential decay for four activation and spectral-radius combinations. — Gradient norm magnitude as gradients propagate backward through 60 timesteps for four RNN configurations. The sigmoid configurations decay by many orders of magnitude because sigmoid's maximum derivative is 0.25, while tanh preserves substantially more signal. A spectral radius closer to 1 slows the decay for either activation.

The plot confirms the theory. Even the most favorable configuration, tanh activations with a spectral radius near 1, shows the gradient norm decaying by several orders of magnitude over 60 timesteps. The sigmoid variants collapse far faster. When training on sequences of length 100 or more, the gradients reaching early timesteps are negligibly small: the weights responsible for long-range dependencies receive near-zero updates throughout training.

Notice how the y-axis of this plot spans many orders of magnitude. The gradient norm at step 60 is often a billion or a trillion times smaller. When you print out gradient values during training and see something like $3.2 \times 10^{-11}$ , that is not a rounding error. That is the actual gradient, and it is telling you that the network is completely blind to any influence from 60 steps ago.