Vanishing Gradients › The Path to LSTM: What the Architecture Needs

The vanishing gradient problem has practical consequences. It is the reason vanilla RNNs are rarely used in modern NLP despite their elegant formulation. The failure is architectural: there is no mechanism in the basic RNN to independently control how much information is retained versus discarded across time, or how much gradient signal is allowed to flow backward.

Any solution to the vanishing gradient problem must address this root cause. The architecture needs a way to:

• Preserve information over long distances without the information being transformed (and potentially diluted) at every step.
• Selectively update which information in memory is relevant to the current input.
• Control gradient flow such that gradients can propagate backward without being repeatedly multiplied by the same shrinking factors.

The LSTM architecture, introduced by Hochreiter and Schmidhuber in 1997, addresses all three requirements through a combination of a separate memory cell and learnable gate mechanisms. Rather than passing the hidden state through a nonlinearity at every step, the LSTM maintains a cell state $c_t$ that can change by addition rather than multiplication. Because addition does not shrink gradient magnitudes the way multiplication does, gradients can flow backward through the cell state pathway with minimal attenuation.

To see intuitively why addition helps, consider the gradient through an additive update. Suppose the cell state evolves as $c_t = c_{t-1} + f_t$ for some input-dependent function $f_t$. The gradient of $c_T$ with respect to $c_t$ is then:

$$\frac{\partial c_T}{\partial c_t} = \frac{\partial}{\partial c_t} \left( c_t + \sum_{k=t+1}^{T} f_k \right) = 1$$

The gradient is identically 1, regardless of sequence length. Information from any timestep is equally accessible via the cell state pathway. The LSTM's actual update is slightly more complex because the forget gate modulates $c_{t-1}$ before the addition, but the key insight remains: an additive update path creates a near-constant gradient that does not decay with distance.

This is the conceptual heart of LSTM and, later, of residual connections in deep networks. The additive pathway is a gradient highway, allowing the loss at the final timestep to reach and update weights at any earlier position. The gates in LSTM control what information enters the highway and what exits it, but the highway itself is always open.

We will explore the LSTM architecture in detail in the next chapter, and then examine the mathematics of LSTM gradient flow in the chapter after that. The key insight to carry forward from this chapter is that the vanishing gradient problem is a consequence of the multiplicative gradient chain, and any solution must break or bypass this chain for long-distance signals.