Long Short-Term Memory (1997) › The Path to Transformers

The limitations of LSTMs, particularly their sequential processing bottleneck and difficulty with very long sequences, motivated researchers to explore fundamentally different approaches to sequence modeling. The architecture that would eventually supersede LSTMs took a radically different philosophical approach to the problem of maintaining context across sequences.

Rather than processing sequences element by element while maintaining an evolving memory state, transformer architectures introduced in 2017 allowed every position in a sequence to directly attend to every other position in parallel. Through the attention mechanism, a model could dynamically determine which parts of the input were relevant to processing each position, computing these relevance weights and using them to combine information from across the entire sequence. This eliminated the sequential bottleneck that limited LSTM parallelization and avoided the need to compress long-range dependencies through a fixed-size memory state.

The architectures took fundamentally different approaches. LSTMs attempted to solve the long-range dependency problem by creating better memory mechanisms, finding ways to carry information forward through time. Transformers instead abolished the notion of processing through time at all, treating the sequence as a set of positions that could all be processed simultaneously with learned patterns of attention determining how information flowed between positions.

This shift brought enormous practical advantages. Transformers could be trained much more efficiently on modern hardware because their computations could be fully parallelized across sequence positions. They scaled more effectively to longer sequences because every position had direct access to every other position rather than information having to flow through sequential state updates. They could learn more complex dependencies because the attention mechanism could form direct connections between distant elements rather than relying on information to survive passage through many intermediate states.

Yet the transition from LSTMs to transformers was not a complete rejection of LSTM principles. The attention mechanism itself can be viewed as a form of dynamic, learned gating that determines what information is relevant at each position. The multi-head attention in transformers allows different attention heads to specialize in different types of dependencies, analogous to how different dimensions of the LSTM cell state could carry different types of information. The fundamental insight that neural networks need explicit mechanisms for selective information routing, established by LSTMs, carried forward even as the specific implementation changed.