LSTM Gradient Flow › When LSTMs Still Need Gradient Clipping

The constant error carousel dramatically improves gradient flow through the cell state, but it does not eliminate all gradient problems in LSTMs. Exploding gradients remain a concern, and gradient clipping (as covered in the Gradient Clipping chapter of Part X) is still commonly used.

While the cell state path is protected by the forget gate, the hidden state path is not. The hidden state $\mathbf{h}_t = \mathbf{o}_t \odot \tanh(\mathbf{c}_t)$ depends on the output gate and the cell state. Gradients flowing through the hidden state path encounter weight matrices ($\mathbf{W}_o, \mathbf{U}_o$) whose spectral radii are not bounded below 1.

In practice, the combined effect of cell state and hidden state gradient paths is that vanishing gradients are largely mitigated by the cell state path (as long as forget gate values are reasonable), while exploding gradients remain possible through the hidden state path when weight matrices have large singular values.

Gradient clipping (typically clipping the global gradient norm to a threshold like 1.0 or 5.0) addresses the exploding gradient problem without interfering with the long-range learning enabled by the cell state path. The clipping operation truncates large gradient vectors to lie within a ball of fixed radius, preventing runaway updates while preserving the direction of the gradient.

To see why clipping is still necessary, consider what happens when the LSTM encounters an unusual input sequence. Even with a well-trained forget gate, certain inputs can trigger large activations in the output gate or the candidate cell state, leading to a spike in the gradient through the hidden state path. Without clipping, this spike could destabilize training by causing a very large parameter update that undoes previously learned behavior. With clipping, the spike is attenuated, and training continues smoothly.

Gradient clipping should generally be applied when training LSTMs on:

• Long sequences (T > 50) where rare events can cause occasional large gradients
• Tasks with variable-length sequences where long sequences dominate the gradient distribution
• Networks with large hidden dimensions (h > 512) where weight matrices are large
• Early in training before weights have stabilized

A practical rule: start with gradient clipping enabled (norm threshold 1.0) and monitor gradient norms during training. If gradient norms are consistently below the threshold (the clip never activates), you can remove it. If clipping activates frequently, the model may have an architectural issue or a learning rate that's too high.

During LSTM training, monitoring gradient health helps catch problems early. Several metrics are useful:

The gradient norm (before and after clipping) tells you how often clipping activates and whether the network's effective gradient magnitude is shrinking over time. A gradient norm that slowly trends downward during training can indicate the onset of vanishing gradients even in an LSTM, especially if the forget gate values have not yet been pushed toward 1.

The forget gate mean across the hidden dimensions tells you whether the network is learning to preserve information. A mean below 0.3 suggests the network is forgetting most things at each step, which is problematic for long-range tasks. If you observe this, consider increasing the forget gate bias or reducing the learning rate for the gate parameters.

The cell state variance across a batch tells you whether cell states are collapsing toward a constant or diverging toward very large values. Healthy cell states should have moderate variance that remains stable through training. A collapsing cell state variance often indicates that the forget gate is stuck near 0, while an exploding cell state variance suggests the input gate is consistently producing large values without adequate forgetting.

Combining these diagnostics into a training monitoring loop takes only a few extra lines of code and can save hours of debugging when something goes wrong.