Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Gradient Clipping › The Exploding Gradient Problem

Michael BrenndoerferUpdated April 24, 20259 min read

Language AI HandbookAll chapters

Part 10 of 67Neural Network Foundations

Chapter 73Gradient Clipping › The Exploding Gradient Problem

Reading section 2 of 14 · The Exploding Gradient Problem · 9 minSection 2 of 14 · 9 min

Section 2 of 14 · 9 min read

Gradient Clipping › The Exploding Gradient ProblemOne section · Sections in this chapter2 / 14 ↓

Sections in this chapter14

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

During backpropagation, gradients flow backward through the network, accumulating contributions from each layer. In deep networks or recurrent architectures, this can create a feedback loop where gradients compound multiplicatively. To understand why, consider what backpropagation computes at each layer: the chain rule tells us to multiply the downstream gradient by the local Jacobian matrix. In a sequence of $n$ layers, the gradient must pass through $n$ such multiplications before reaching the first layer's parameters. Each multiplication can scale the gradient up or down depending on the magnitude of the Jacobian.

The problem becomes acute in recurrent networks because the same weight matrix is applied at every timestep. This means the gradient must flow backward through potentially hundreds of identical multiplications by the same matrix. Unlike a feedforward network where each layer has its own independent weights, an RNN has a single recurrence matrix whose spectral properties determine whether gradients grow or shrink throughout the entire backpropagation pass.

Exploding Gradients

Exploding gradients occur when gradient magnitudes grow exponentially during backpropagation. This happens when the chain rule produces products of values greater than 1 that compound across many layers, resulting in weight updates so large they destabilize training.

Consider a simple recurrent network processing a sequence of $T$ timesteps. At each step, the gradient is multiplied by the weight matrix $\mathbf{W}$ . If the largest eigenvalue of that matrix exceeds 1, gradients grow exponentially. After $T$ multiplications, the gradient magnitude scales as:

\|\mathbf{g}_T\| \approx \|\mathbf{g}_0\| \cdot \lambda_{\max}^T

where:

$\|\mathbf{g}_0\|$ : the initial gradient magnitude at the output layer
$\|\mathbf{g}_T\|$ : the gradient magnitude after backpropagating through $T$ timesteps
$\lambda_{\max}$ : the largest eigenvalue of the weight matrix (or its spectral norm)
$T$ : the number of timesteps (or layers) the gradient passes through

With $T = 100$ timesteps and $\lambda_{\max} = 1.1$ , the gradient grows by a factor of $1.1^{100} \approx 13{,}781$ . Even this modest 10% amplification per step compounds into a catastrophic explosion. The key insight is that the amplification is multiplicative, not additive. Each step does not add a little bit to the gradient; it multiplies the gradient by a factor, so even small factors above 1.0 lead to exponential growth over long sequences.

Notice that this formulation assumes the eigenvalue is constant across timesteps, which is exactly what happens in recurrent networks where the same weight matrix is applied repeatedly. In feedforward networks, each layer has its own Jacobian, and the product of many different Jacobians can still explode, but it requires a more consistent directional alignment of the growth. Recurrent networks are especially susceptible precisely because the same amplification factor is applied over and over.

In[3]:

Code

# Simulating gradient growth in a recurrent network
timesteps = 100
growth_factor = 1.1  # Gradient multiplier per step

# Track gradient magnitude over time
gradient_magnitudes = [1.0]  # Start with unit gradient
for t in range(timesteps):
    gradient_magnitudes.append(gradient_magnitudes[-1] * growth_factor)

Out[4]:

Console

Gradient Growth Over Timesteps
--------------------------------------------------
Initial gradient magnitude: 1.00
After 10 steps: 2.59
After 50 steps: 117.39
After 100 steps: 13780.61
Growth factor: 13781x

Out[5]:

Visualization

Line plot showing exponential growth curve of gradient magnitude from 1 to nearly 14000 over 100 timesteps on linear scale. — Gradient magnitude on a linear scale showing the characteristic hockey-stick curve of exponential growth. The gradient stays near zero for most of the sequence before skyrocketing in the final timesteps.

Line plot showing exponential growth on log scale as a straight line with annotations at key points. — Gradient magnitude on a logarithmic scale confirming exponential growth with a straight line, with annotations showing 117x amplification at step 50 and 13,781x at step 100.

The exponential growth is striking. With just a 10% amplification per step, the gradient grows nearly 14,000 times larger by the end of the sequence. The linear scale plot makes this viscerally clear: the gradient is essentially invisible for most of the sequence, then explodes in the final timesteps. The log scale plot reveals the uniform, relentless nature of the growth: a perfectly straight line on a logarithmic axis, confirming that every step multiplies the gradient by the same factor.

In practice, these explosive gradients lead to NaN losses, parameter values shooting to infinity, and complete training failure. The model's weights get updated by amounts that can be thousands of times larger than the entire weight value itself, pushing parameters into regions of parameter space so extreme that the network cannot recover. If you have ever seen a training loss suddenly jump to NaN or watched a gradient norm histogram spike into millions, you have witnessed a gradient explosion firsthand.

Detecting Gradient ExplosionsLink Copied

Before clipping, you need to know when gradients are problematic. The clearest signal is the gradient norm, which measures the overall magnitude of gradients across all parameters. For a model with parameters $\theta_1, \theta_2, \ldots, \theta_n$ , each with gradient $\nabla_{\theta_i} L$ , the global L2 norm is:

\|\nabla L\|_2 = \sqrt{\sum_{i=1}^{n} \|\nabla_{\theta_i} L\|_2^2}

where:

$\nabla_{\theta_i} L$ : the gradient of the loss $L$ with respect to parameter $\theta_i$ (a tensor)
$\|\nabla_{\theta_i} L\|_2$ : the L2 norm of that parameter's gradient
$\|\nabla L\|_2$ : the global gradient norm across all parameters

This single scalar summarizes the entire gradient's magnitude, making it easy to monitor and compare across training steps. Think of the gradient norm as a speedometer for your optimizer: normal training keeps the needle in a reasonable range, while gradient explosions send it flying off the scale.

The global norm is also valuable because it aggregates information from all parameters simultaneously. Rather than tracking thousands of individual gradient tensors, you have one number that tells you whether the entire gradient update is within a safe range. When this number spikes dramatically above its usual baseline, that is your signal to investigate, and it is exactly what gradient clipping will automatically handle.

In[6]:

Code

import torch
import torch.nn as nn

# Create a simple network to demonstrate gradient monitoring
model = nn.Sequential(
    nn.Linear(10, 64), nn.ReLU(), nn.Linear(64, 64), nn.ReLU(), nn.Linear(64, 1)
)

# Compute gradients for a random batch
x = torch.randn(32, 10)
y = torch.randn(32, 1)

loss = nn.MSELoss()(model(x), y)
loss.backward()


def compute_gradient_norm(model):
    """Compute the global L2 norm of all gradients."""
    total_norm = 0.0
    for param in model.parameters():
        if param.grad is not None:
            total_norm += param.grad.data.norm(2).item() ** 2
    return total_norm**0.5

Out[7]:

Console

Total gradient norm: 1.1497

Per-layer gradient norms:
  0.weight: 0.2031 (3.1% of total)
  0.bias: 0.0752 (0.4% of total)
  2.weight: 0.6288 (29.9% of total)
  2.bias: 0.1762 (2.3% of total)
  4.weight: 0.7905 (47.3% of total)
  4.bias: 0.4730 (16.9% of total)

The global gradient norm for this network is relatively modest, typical for a well-initialized shallow network on random data. The per-layer breakdown shows how each parameter contributes to the total. Notice that the contributions are not evenly distributed: some layers dominate the total norm while others contribute relatively little. In practice, you would monitor this norm over training batches to detect sudden spikes, which signal potential gradient explosions. Many practitioners log the gradient norm to a dashboard like TensorBoard or Weights and Biases and set up alerts for values exceeding a threshold.

What Causes Gradient Explosions?Link Copied

Several architectural and training choices make gradient explosions more likely. Understanding these root causes helps you anticipate when clipping will be necessary before you see training fail.

The most fundamental cause is the depth of the computational graph. More layers mean more gradient multiplications. Each layer's Jacobian contributes to the product, and if many Jacobians have spectral norms greater than 1, gradients grow exponentially as they backpropagate toward earlier layers. This is purely a consequence of the chain rule: derivatives multiply, and multiplying many numbers greater than 1 together produces large numbers.

Recurrent networks are especially vulnerable because they process sequences by repeatedly applying the same weights. A sequence of 200 tokens through an RNN creates a computational graph 200 layers deep, but with only one set of weights shared across all those layers. The gradient must flow back through all 200 steps, multiplied by the same weight matrix each time. If the spectral radius of that matrix exceeds 1, the gradient explosion is inevitable regardless of how well the network is otherwise designed.

Several additional factors make explosions more likely:

Large learning rates: Even moderate gradients become destructive updates with high learning rates. The explosion may be in the update magnitude rather than the gradient itself.
Improper initialization: Weight initialization that starts parameters too large can push networks into explosive regimes from the first forward pass.
Long sequences: For RNNs and LSTMs, longer sequences create deeper computational graphs, compounding any per-step amplification.
Sparse, extreme data: Unusual data points with very large feature values can trigger large losses that backpropagate as large gradients throughout the network.

Understanding these causes is important because it reveals that gradient clipping is a preventive measure, not a cure. The underlying instability often persists even with clipping; you are simply preventing it from derailing training. If explosions happen constantly despite clipping, revisiting architecture design, initialization, or learning rates may be necessary.