Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Vanishing Gradients › The Geometry of the Problem: A Worked Example

Michael BrenndoerferMay 8, 20253 min read

Language AI HandbookAll chapters

Part 11 of 67Recurrent Neural Networks

Chapter 76Vanishing Gradients › The Geometry of the Problem: A Worked Example

Reading section 9 of 18 · The Geometry of the Problem: A Worked Example · 3 minSection 9 of 18 · 3 min

Section 9 of 18 · 3 min read

Vanishing Gradients › The Geometry of the Problem: A Worked ExampleOne section · Sections in this chapter9 / 18 ↓

Sections in this chapter18

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

To make the vanishing gradient concrete, let's trace a specific gradient through a small RNN by hand.

Consider an RNN with hidden size 2 and a single timestep transition matrix:

W_h = \begin{pmatrix} 0.5 & 0.3 \\ 0.2 & 0.4 \end{pmatrix}

The spectral radius of this matrix is approximately 0.71 (less than 1). Now imagine a sequence of length 5. The gradient of $L$ with respect to $h_0$ involves multiplying through 5 Jacobian factors:

\frac{\partial L}{\partial h_0} = \frac{\partial L}{\partial h_5} \cdot \prod_{k=1}^{5} J_k

where each $J_k = \text{diag}(\tanh'(z_k)) \cdot W_h$ has spectral radius at most 0.71.

After 5 steps: $0.71^5 \approx 0.18$ . The gradient has shrunk by a factor of about 5.

After 10 steps: $0.71^{10} \approx 0.034$ . A factor of 30 reduction.

After 20 steps: $0.71^{20} \approx 0.0011$ . A factor of 900 reduction.

After 40 steps: $0.71^{40} \approx 1.2 \times 10^{-6}$ . Numerically negligible.

This is just one specific matrix. Let's compute this numerically to see how it plays out across various sequence lengths:

In[6]:

Code

W_example = np.array([[0.5, 0.3], [0.2, 0.4]])
eigvals_example = np.linalg.eigvals(W_example)
spectral_radius_example = np.max(np.abs(eigvals_example))

sequence_lengths = [5, 10, 20, 40, 60]

np.random.seed(123)
h_ex = np.array([0.5, -0.3])
grad_ex = np.eye(2)
step_norms = [np.linalg.norm(grad_ex, "fro")]

for step in range(max(sequence_lengths)):
    z_ex = W_example @ h_ex
    d_ex = tanh_deriv(z_ex)
    J_ex = np.diag(d_ex) @ W_example
    grad_ex = J_ex @ grad_ex
    h_ex = np.tanh(z_ex)
    step_norms.append(np.linalg.norm(grad_ex, "fro"))

grad_norms_by_length = {
    length: step_norms[length] for length in sequence_lengths
}

Out[7]:

Console

Spectral radius of W_h: 0.7000

Sequence length      Gradient norm        Relative to step 1       
-----------------------------------------------------------------
5                    1.679169e-01         2.322032e-01             
10                   2.821764e-02         3.902066e-02             
20                   7.970731e-04         1.102230e-03             
40                   6.360027e-07         8.794940e-07             
60                   5.074810e-10         7.017682e-10

The output shows the gradient norm collapsing exponentially. By step 40, it has shrunk by more than 6 orders of magnitude relative to step 5. Any weight update proportional to this gradient is effectively zero regardless of learning rate choice.

The numbers make the problem viscerally clear. A learning rate of 0.01 applied to a gradient of $10^{-7}$ produces a weight update of $10^{-9}$ . With 32-bit floating-point precision, this update rounds to zero. Long-range dependencies are simply unlearnable.

What the Worked Example Reveals About InitializationLink Copied

The example above used a specific matrix with spectral radius 0.71. But the choice of initialization matters enormously for how quickly gradients vanish. Early practitioners tried to fight vanishing gradients by using larger initialization, hoping to push the spectral radius closer to 1. This helps, but only partially.

The issue is that the tanh nonlinearity still contributes damping factors less than 1 at every step (unless all pre-activations are exactly at zero). Even with a weight matrix whose spectral radius is exactly 1, the effective spectral radius of the full Jacobian $\text{diag}(\tanh'(z_k)) \cdot W_h$ will be less than 1 at every step where the hidden state has moved away from zero. And as the network processes information, the hidden states will inevitably move away from zero.

This suggests that careful initialization can delay but not eliminate vanishing gradients. The network might successfully learn dependencies up to 30 or 40 steps with good initialization, compared to 10 or 15 steps with poor initialization. But for sequences of length 100, 500, or 1000, even the best initialization leaves most of the sequence out of the effective gradient range.