Recurrent Neural Networks (1995) › The Limitations That Spurred Further Innovation

Despite their success in bringing memory to neural networks, RNNs revealed fundamental limitations that became increasingly apparent as researchers pushed them to handle longer sequences and more complex tasks. These limitations were not minor engineering challenges but rather deep problems rooted in the architecture itself. Understanding these issues is essential to appreciating why later innovations like LSTMs and GRUs were necessary, and why the field eventually moved toward attention mechanisms and transformers.

The most serious problem facing RNNs is known as the vanishing gradient problem. To understand this issue, recall that RNNs are trained using backpropagation through time, which involves computing gradients by flowing error signals backward through the unfolded network. At each time step, the gradient is multiplied by the weight matrix and by the derivative of the activation function. When this multiplication is repeated many times across a long sequence, the gradient can shrink exponentially if the weight matrix has eigenvalues less than one or if the activation function has small derivatives. As a result, the gradient signal that reaches the early time steps becomes vanishingly small, making it nearly impossible for the network to learn long range dependencies. In practice, this means that an RNN might struggle to learn that the first word of a sentence is relevant for predicting the last word, even if that relationship is important for the task at hand.

The vanishing gradient problem is particularly severe for activation functions like the logistic sigmoid or tanh, which have derivatives that are less than one over most of their range. When many such derivatives are multiplied together, the product shrinks toward zero. This effect becomes more pronounced as sequences get longer, limiting the effective memory span of the RNN to only a few time steps. While the hidden state in principle allows the network to remember arbitrarily long contexts, in practice the gradient signal needed for learning fades before it can propagate across long sequences, preventing the network from learning to use that long range information.

Related to the vanishing gradient problem is the issue of exploding gradients, where the gradient grows exponentially during backpropagation. This happens when the weight matrix has large eigenvalues or when derivatives accumulate in a way that causes the gradient to blow up. Exploding gradients can destabilize training, causing the network's parameters to make wild updates that degrade performance. A common solution is gradient clipping, which caps the magnitude of the gradient to prevent these extreme updates. While gradient clipping is effective at managing explosions, it does nothing to solve the vanishing gradient problem, which remains the more fundamental obstacle.

Another limitation is the fixed size of the hidden state. The hidden state is a vector of fixed dimensionality, and it must encode all of the relevant information from the entire sequence up to the current time step. As sequences get longer, the amount of information that needs to be summarized grows, but the capacity of the hidden state does not. This creates an information bottleneck, where the network is forced to compress more and more context into the same amount of space. In practice, this means that older information tends to be overwritten by newer information, and the network's effective memory is limited even if it is not constrained by vanishing gradients.

These limitations became increasingly problematic as researchers attempted to apply RNNs to tasks that required understanding long documents, maintaining context over extended dialogues, or learning dependencies that span many time steps. The recognition of these problems motivated the development of more sophisticated architectures that could address them, leading directly to the innovations that would follow in the next phase of the field's evolution.