Vanishing Gradients › Attempts to Mitigate Vanishing Gradients Without Architecture Changes

Before the LSTM became the standard solution, several techniques were proposed to mitigate vanishing gradients without fundamentally changing the architecture. Understanding why these partial solutions fail is instructive.

One approach was to initialize $W_h$ to be close to the identity matrix or to have spectral radius exactly 1. This is called identity initialization or orthogonal initialization. With an identity-initialized $W_h$, the gradient product starts at the identity and decays only due to the nonlinearity damping factors. This delays vanishing but does not prevent it.

The practical limitation is that the weight matrix must change during training to learn useful representations. As soon as weight updates begin, the spectral radius moves away from 1, and gradient decay resumes. Constraining the spectral radius during training requires projecting the weight matrix onto the set of orthogonal matrices after every update, which is computationally expensive and disrupts the optimization landscape.

Irene Sutskever and colleagues experimented with careful initialization strategies in the early 2010s and found that they could improve the performance of vanilla RNNs on certain tasks. But the gains were limited: a well-initialized vanilla RNN could match an LSTM on short-sequence tasks but still failed on long sequences where the gradient decay problem was fundamental.

A pragmatic workaround is truncated BPTT: instead of propagating gradients through the full sequence, you stop the backward pass after $k$ steps. This makes training faster and avoids the numerical instability of very long gradient chains, but it does not solve the problem. It simply admits defeat by explicitly limiting the temporal horizon. The network cannot learn dependencies longer than $k$ steps by construction.

Truncated BPTT is still widely used as an optimization technique even with LSTM networks, because it reduces memory usage and improves training speed. With LSTMs, the cell state provides a separate gradient path that persists even when the hidden state gradient is truncated, so truncated BPTT is less damaging in practice than it is for vanilla RNNs.

The truncation length $k$ is a hyperparameter that creates a tradeoff between computational cost and temporal range. Setting $k$ to 20 means the network can potentially learn dependencies up to 20 steps, but no further. Setting $k$ to 100 allows longer dependencies but requires holding 100 steps of activations in memory and computing 100 backward steps per update, which is expensive. Most practitioners using vanilla RNNs chose small $k$ values to keep training tractable, which inadvertently reinforced the impression that RNNs are fundamentally local models.

Gradient clipping addresses the exploding gradient problem but has no effect on vanishing gradients. Clipping prevents the gradient from becoming too large, but it cannot inflate a gradient that has already shrunk to near zero. The two problems are not symmetric.

Some practitioners mistakenly apply gradient clipping hoping to stabilize training when they are experiencing vanishing gradients. The loss may still appear to decrease (because local, short-range updates are still occurring), and the absence of NaN errors suggests training is healthy. The silent failure mode of vanishing gradients means these practitioners may train for a long time without realizing that long-range dependencies are never being learned.

A more radical approach is to abandon gradient-based learning for the recurrent weights entirely. Echo State Networks (ESNs) and Liquid State Machines fix the recurrent weights randomly and only train the output layer via linear regression. This completely sidesteps the vanishing gradient problem by eliminating the recurrent gradient chain. The fixed recurrent weights create a complex dynamical system (the "reservoir") whose state encodes the recent history, and the output layer learns to read out the relevant information.

ESNs can capture some long-range dependencies through the dynamics of the reservoir, but the reliance on random weights limits their flexibility. The reservoir must be large enough and the dynamics complex enough to implicitly represent all the dependencies the task requires, which is not guaranteed. ESNs found niche applications but never became the dominant approach to sequence modeling.