LSTM Gradient Flow › Quantitative Gradient Norm Analysis Over Time
Language AI HandbookAll chapters LSTM Gradient Flow › Quantitative Gradient Norm Analysis Over TimeOne section · Sections in this chapter7 / 17 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Rather than reasoning about gradient flow in the abstract, we can measure it directly. Comparing the empirical gradient norms for vanilla RNNs and LSTMs trained on a simple sequence task reveals the theoretical differences in striking quantitative terms.
Setting Up the Measurement
A common approach is to initialize both networks with the same architecture size, train on identical data for a small number of steps, and then measure how the gradient norm with respect to the initial hidden state (or cell state for LSTMs) varies as a function of sequence length.
For a sequence of length , the relevant quantity is:
for the RNN, and:
for the LSTM cell state path. As increases, how do these norms change?
import numpy as np
np.random.seed(42)
def sigmoid(x):
return 1 / (1 + np.exp(-np.clip(x, -20, 20)))
def tanh_deriv(x):
return 1 - np.tanh(x) ** 2
def simulate_rnn_gradient_norm(T, hidden_size=32, n_trials=20):
"""
Simulate the expected gradient norm for a vanilla RNN over T timesteps.
Returns mean and std across random initializations.
"""
norms = []
for _ in range(n_trials):
# Random weight matrix normalized to spectral radius 0.9
W = np.random.randn(hidden_size, hidden_size) / np.sqrt(hidden_size)
eigvals = np.linalg.eigvals(W)
W = W * (0.9 / np.max(np.abs(eigvals)))
# Simulate gradient flow
grad = np.ones(hidden_size)
for t in range(T):
z = np.random.randn(hidden_size)
tanh_d = tanh_deriv(z)
grad = W.T @ (tanh_d * grad)
norms.append(np.linalg.norm(grad))
return np.mean(norms), np.std(norms)
def simulate_lstm_gradient_norm(
T, hidden_size=32, forget_bias=1.0, n_trials=20
):
"""
Simulate gradient norm for LSTM cell state path over T timesteps.
forget_bias: initial bias for forget gate (higher = forget gate closer to 1)
"""
norms = []
for _ in range(n_trials):
grad = np.ones(hidden_size)
for t in range(T):
f_logit = forget_bias + 0.3 * np.random.randn(hidden_size)
f = sigmoid(f_logit)
grad = f * grad
norms.append(np.linalg.norm(grad))
return np.mean(norms), np.std(norms)
# Measure across different sequence lengths
lengths = [5, 10, 20, 30, 50, 75, 100]
rnn_means, rnn_stds = [], []
lstm_means, lstm_stds = [], []
for T in lengths:
mu, sig = simulate_rnn_gradient_norm(T)
rnn_means.append(mu)
rnn_stds.append(sig)
mu, sig = simulate_lstm_gradient_norm(T, forget_bias=1.0)
lstm_means.append(mu)
lstm_stds.append(sig)
rnn_means = np.array(rnn_means)
rnn_stds = np.array(rnn_stds)
lstm_means = np.array(lstm_means)
lstm_stds = np.array(lstm_stds)Gradient norm comparison (mean +/- std across 20 trials):
Length RNN norm LSTM norm
----------------------------------------
5 0.3473+/-0.2105 1.1730+/-0.0348
10 0.0241+/-0.0147 0.2416+/-0.0131
20 0.0001+/-0.0001 0.0101+/-0.0004
30 0.0000+/-0.0000 0.0004+/-0.0000
50 0.0000+/-0.0000 0.0000+/-0.0000
75 0.0000+/-0.0000 0.0000+/-0.0000
100 0.0000+/-0.0000 0.0000+/-0.0000The table shows the gradient norm decaying rapidly for the vanilla RNN as sequence length increases, while the LSTM cell state retains a substantially larger gradient norm at the same sequence lengths. This quantitative difference translates directly into the LSTM's ability to learn long-range dependencies.
