LSTM Architecture › LSTM Architecture

In the previous chapters, we saw how vanilla RNNs struggle with long sequences. As we established in the Vanishing Gradients chapter, gradients either vanish into insignificance or explode beyond control, making it nearly impossible to learn dependencies that span more than a few dozen timesteps. This limitation motivated researchers to rethink the fundamental architecture of recurrent networks.

Long Short-Term Memory (LSTM) networks, introduced by Hochreiter and Schmidhuber in 1997, solve the vanishing gradient problem through a deceptively simple architectural insight: instead of forcing all information to pass through repeated nonlinear transformations at every timestep, create a dedicated pathway where information can flow relatively unchanged across many timesteps. This "memory highway" allows LSTMs to maintain and access information over hundreds or even thousands of steps, enabling applications that were previously impossible with vanilla RNNs. The key idea is so elegant that it can be stated in a single sentence: separate the cell's memory from its output, and let the memory update additively rather than through repeated multiplication.