LSTM Gate Equations › Limitations and Practical Considerations

LSTMs represented a major breakthrough in sequence modeling, but they come with important limitations that practitioners should understand.

Computational Cost: Each time step requires sequential computation because the hidden state at time $t$ depends on the hidden state at time $t-1$. This sequential dependency prevents parallelization across time steps, making LSTMs slow to train on long sequences. A 1000-step sequence requires 1000 sequential operations, regardless of how many GPUs you have. This is the fundamental limitation that motivated the development of Transformers, which can process all positions in parallel.

Memory Requirements: LSTMs must store hidden and cell states for every time step during training to enable backpropagation through time. For a sequence of length $T$ with hidden dimension $h$, this requires $O(T \cdot h)$ memory per layer, where $T$ is the number of time steps and $h$ is the hidden state size. With long sequences and deep networks, memory can become a bottleneck. Techniques like truncated backpropagation through time can help, but they sacrifice gradient accuracy for long-range dependencies.

Gradient Flow: Despite the cell state's "information highway," gradients still face challenges over very long sequences. The forget gate must maintain values very close to 1 for hundreds of time steps to preserve gradients, which is difficult to learn. In practice, LSTMs struggle with dependencies beyond a few hundred time steps.

Hyperparameter Sensitivity: The hidden dimension, number of layers, learning rate, and gradient clipping threshold all interact in complex ways. LSTMs can be finicky to train, often requiring careful tuning. The forget gate bias initialization (setting it to 1 or 2) is one example of a trick that significantly affects training stability.

Despite these limitations, LSTMs remain valuable in many contexts. They are well-suited for streaming applications where you process data one element at a time. They work well for moderate-length sequences (up to a few hundred tokens). They also require less data to train than Transformers for smaller-scale problems. Understanding their equations deeply, as we've done in this chapter, helps you recognize when they're the right tool and when to reach for alternatives.