Vanishing Gradients › Historical Context: Why This Problem Went Unsolved for So Long

The vanishing gradient problem was not discovered suddenly. It accumulated gradually through the late 1980s and early 1990s as researchers attempted to train recurrent networks on increasingly complex sequence tasks. Sepp Hochreiter identified and formalized the problem in his 1991 diploma thesis. This provides the mathematical analysis we have largely covered in this chapter. The formal analysis in English attracted relatively little attention until the LSTM paper in 1997.

Part of the reason the problem persisted without a reliable solution for several years was that exploding gradients received more attention. Exploding gradients were easy to observe and directly fixable with clipping. Vanishing gradients were difficult to observe and required an architectural solution rather than a training trick.

Another factor was the prevailing belief that training issues with RNNs were due to poor optimization, insufficient data, or bad initialization rather than a fundamental architectural flaw. The culture of the field emphasized that neural networks were universal approximators and that with the right training setup, any network could learn any function. The idea that an architecture could be provably limited in what it could learn via gradient descent, even if theoretically capable of representing the target function with the right weights, was subtle and not immediately accepted.

Hochreiter and Schmidhuber's 1997 LSTM paper was the turning point. It provided the theoretical analysis and a complete architectural solution that was empirically validated on the exact tasks where vanilla RNNs failed. The success of LSTMs on long-range dependency benchmarks demonstrated that the vanishing gradient problem was the bottleneck and that architectural changes could overcome it.

The two-decade gap between the RNN's invention and the LSTM's adoption illustrates a general pattern in deep learning: fundamental limitations are often recognized empirically before they are understood theoretically, and solutions often precede complete theoretical understanding of why they work. The gradient flow analysis we have covered in this chapter was largely worked out by Hochreiter in 1991, but it took the empirical success of LSTMs to convince the broader community that the analysis was pointing to a real and solvable problem.