LSTM Gradient Flow › Vanishing Gradients: Why LSTM Avoids Them

The LSTM does not completely eliminate the vanishing gradient problem; it creates a path where vanishing is far less likely. To understand this precisely, we need to distinguish between the different gradient paths through an LSTM.

When we backpropagate through an LSTM, gradients flow through multiple paths simultaneously. Consider the gradient flowing from timestep $T$ back to timestep $k$. There are several routes:

The cell state path flows directly through the cell state with only forget gate multiplications. As derived above, this path has the gradient:

$$\frac{\partial L}{\partial \mathbf{c}_k} \approx \frac{\partial L}{\partial \mathbf{c}_T} \odot \mathbf{f}_{k+1} \odot \mathbf{f}_{k+2} \odot \cdots \odot \mathbf{f}_T$$

The hidden state path flows through the hidden states, passing through the output gate and the tanh nonlinearity at each step, in addition to the cell state dynamics. This path is much more complex and does experience more gradient attenuation.

The gate paths flow through the individual gate computations (forget, input, output, candidate), each involving sigmoid or tanh nonlinearities and weight matrices.

The key insight is that during optimization, the gradient flowing through the cell state path provides the essential long-range credit assignment signal. The other paths contribute additional gradient information for nearby timesteps. Together, they enable the LSTM to learn both long-range and short-range dependencies. You can think of the cell state path as the "slow lane" for long-distance gradient traffic, while the hidden state path is the "fast lane" for short-range gradient adjustment.

For the LSTM cell state path, the gradient vanishes when:

$$\left\| \prod_{t=k+1}^{T} \mathbf{f}_t \right\| \approx 0$$

This happens when forget gate values are consistently small across many timesteps. Unlike vanilla RNNs, the forget gate values are learned, and the network can be trained to keep them near 1 when long-range memory is needed. The vanilla RNN has no such control mechanism; its gradient multipliers are determined by fixed weight matrices applied to potentially varying inputs, with no learned switch to protect gradient flow.

Additionally, because the cell state path involves only element-wise operations (no matrix multiplication), the spectral radius concern that drives vanilla RNN gradient explosions and vanishing is absent. There is no matrix whose eigenvalues need to be tuned; there are only per-dimension forget gate values between 0 and 1.

The LSTM does not solve all gradient problems. The gradient flowing through the hidden state path still experiences attenuation. The output gate tanh nonlinearity has a maximum derivative of 1 but typically much smaller values. The gradient path from the hidden state through the output gate to the cell state introduces multiplicative factors less than 1.

If the forget gate learns to produce very small values, the cell state path itself can vanish. In sequence modeling tasks where inputs frequently reset the relevant context, this is appropriate behavior. But if the network incorrectly learns small forget gate values for dimensions that should carry long-range information, it will fail to learn those dependencies.

The LSTM provides the architectural possibility of vanishing-free gradient flow. Whether that possibility is realized depends on the training dynamics, initialization, and the structure of the target task.