Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Vanishing Gradients › The Vanishing vs Exploding Tradeoff

Michael BrenndoerferMay 8, 20253 min read

Language AI HandbookAll chapters

Part 11 of 67Recurrent Neural Networks

Chapter 76Vanishing Gradients › The Vanishing vs Exploding Tradeoff

Reading section 10 of 18 · The Vanishing vs Exploding Tradeoff · 3 minSection 10 of 18 · 3 min

Section 10 of 18 · 3 min read

Vanishing Gradients › The Vanishing vs Exploding TradeoffOne section · Sections in this chapter10 / 18 ↓

Sections in this chapter18

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The vanishing and exploding problems define the two sides of a knife-edge. When the spectral radius is far below 1, gradients vanish and long-range learning fails. When it is far above 1, gradients explode and training becomes numerically unstable. The stable regime is a narrow band around spectral radius 1, but even within this band, the tanh nonlinearity prevents perfect gradient preservation across long sequences because $|\tanh'(z)| < 1$ for $z \neq 0$ .

This creates a fundamental architectural constraint: vanilla RNNs have no mechanism to maintain gradient magnitude over long distances. The recurrent multiplication is the same every step, driven by the same $W_h$ , and there is no way to independently control gradient flow while also encoding sequence semantics.

In[8]:

Code

spectral_radii = np.linspace(0.1, 1.5, 100)
T_fixed = 50
hidden_size_small = 16

norm_after_T = []
for rho in spectral_radii:
    norms = compute_gradient_norms(
        T_fixed, hidden_size_small, rho, activation="tanh", seed=7
    )
    norm_after_T.append(norms[-1])

norm_after_T = np.array(norm_after_T)

Out[9]:

Visualization

Line plot of gradient norm vs spectral radius showing decay for low rho and explosion for high rho. — Final gradient norm after 50 backward steps as a function of the recurrent weight matrix spectral radius. Spectral radii below 1 cause vanishing gradients with norms collapsing toward zero, while spectral radii above 1 cause explosive growth. The vertical dashed line at rho=1 marks the theoretical boundary, but even slightly above this value, the gradient explodes quickly due to the multiplicative structure.

The plot makes the tradeoff vivid. Below spectral radius 1, gradient norms collapse toward zero over 50 steps. Above spectral radius 1, they explode. The inflection point near 1 appears to offer a solution, but even there, the tanh nonlinearity prevents perfect gradient preservation across long sequences.

The knife-edge at spectral radius 1 is not a stable operating point. Small perturbations to $W_h$ from weight updates during training will push the spectral radius slightly above or below 1, and the gradient behavior will shift accordingly. Maintaining the spectral radius precisely at 1 would require a constraint on the optimization, and such constraints typically slow convergence and limit the expressiveness of the network.