Long Short-Term Memory (1997) › The Architecture of Memory

To understand why LSTMs represented such a significant advance, we must first examine what made traditional RNNs both promising and problematic. The recurrent neural network embodied an elegant idea: maintain a hidden state that summarizes all relevant information from the sequence so far, then update this state at each time step by combining it with the current input. Mathematically, this takes the form:

where $h_t$ represents the hidden state at time $t$, $x_t$ is the current input, and the weight matrices $W_{hh}$ and $W_{xh}$ control how previous state and current input combine. The hyperbolic tangent activation function keeps the hidden state values bounded between -1 and 1.

This architecture works remarkably well for short sequences. If you are predicting the next word given the previous three or four words, an RNN can learn the necessary patterns. The hidden state successfully carries forward the relevant context, and backpropagation through time can assign credit appropriately to the parameters. However, as sequences lengthen, a mathematical problem emerges that ultimately dooms the simple RNN architecture.

During backpropagation, gradients must flow backward through the network, propagating through each time step. At each step, the gradient gets multiplied by the derivative of the hidden state computation, which includes the weight matrix $W_{hh}$. After many time steps, we are effectively multiplying the gradient by this matrix many times. If the largest eigenvalue of this matrix is less than 1, repeated multiplication causes the gradient to shrink exponentially. If the largest eigenvalue exceeds 1, the gradient grows exponentially. In practice, the former case dominates, and gradients vanish, but both scenarios prevent effective learning.

The vanishing gradient problem manifests as an inability to learn long-range dependencies. Suppose a network needs to remember that a sentence began with a singular subject in order to correctly conjugate a verb that appears twenty words later. The error signal from the verb misprediction must propagate back through twenty time steps to adjust the representation of the subject. By the time it arrives, the gradient has become vanishingly small, providing essentially no learning signal. The early time steps remain stuck with their initial random parameters, unable to learn the features that would enable correct long-range predictions.

What made this problem so frustrating for researchers was its fundamental nature. This was not a matter of finding better hyperparameters or training techniques. The mathematical structure of the simple RNN architecture inherently limited what it could learn. Breaking through this barrier would require rethinking how neural networks maintain information over time, introducing architectural mechanisms specifically designed to allow gradients to flow across many time steps without vanishing or exploding.