Vanishing Gradients › Exploding Gradients and Norm Explosion

The complementary problem is exploding gradients. When the spectral radius of $W_h$ exceeds $1/\gamma$, the gradient product grows without bound. In practice, this manifests as the gradient norm increasing by orders of magnitude within a few backward steps.

Exploding gradients are easier to detect than vanishing gradients: the loss suddenly spikes to NaN or infinity, and you can observe the gradient norm growing exponentially in training logs. The standard fix is gradient clipping, covered separately in the Neural Network Foundations chapter.

The asymmetry between vanishing and exploding gradients is important:

• Exploding gradients are detectable and fixable through gradient clipping. The norm becomes absurdly large, giving you a clear signal.
• Vanishing gradients are silent. The loss may appear to decrease, but the network learns only from the most recent few timesteps. There is no error or warning: the weight updates for long-range connections are just approximately zero.

This asymmetry makes vanishing gradients the more dangerous and practically important failure mode.

When gradients explode, the effect is dramatic and unmistakable. A weight update of magnitude $10^5$ to $10^{10}$ is applied to parameters that were initialized with magnitudes near 0.01. The parameters jump to wildly incorrect values, the forward pass produces garbage activations or NaN values, and the loss spikes to infinity. Training collapses.

The fix, gradient clipping, is simple in concept: if the global gradient norm exceeds a threshold (typically 1.0 or 5.0), scale down all gradients proportionally so the global norm equals the threshold. This prevents catastrophic parameter updates while preserving the gradient direction. Gradient clipping does not fix the underlying gradient explosion problem but makes it manageable.

It is worth asking why the same spectral radius condition that causes vanishing gradients for small spectral radii causes explosion for large ones. The answer is that both are consequences of the same product structure. When each factor has spectral radius less than 1, the product converges to zero. When each factor has spectral radius greater than 1, the product diverges. The only difference is the direction of the instability.

In practice, training with exploding gradients is more tractable than training with vanishing gradients, because clipping provides a consistent workaround. Vanishing gradients have no analogous fix: there is no operation you can apply after the fact to recover gradient information that was lost during backpropagation.