Stacked RNNs › Limitations and Impact

Stacked RNNs achieved remarkable results in their time, but they come with significant limitations that motivated the eventual shift to transformers.

The fundamental constraint of any RNN, stacked or not, is that computation is sequential in time. To compute the hidden state at timestep $t$, you need the hidden state at timestep $t-1$. This means you cannot parallelize across the time dimension. For a sequence of length $T$ with $L$ layers, the critical path has length $T \times L$.

Modern accelerators (GPUs and TPUs) achieve peak performance through massive parallelism. Sequential computation underutilizes them. A transformer, by contrast, processes all positions in parallel using attention, allowing it to better exploit modern hardware. This parallelism advantage translates into faster training at the same parameter count.

For a sequence of length 512, a stacked 4-layer LSTM has a sequential critical path of 2048 steps. A transformer with the same depth (4 layers) processes all 512 positions simultaneously, with a critical path of just 4 steps (one per layer). This is a 128x difference in parallelism.

Even with LSTM gating and stacking, learning dependencies that span hundreds or thousands of tokens is challenging. The LSTM cell state does mitigate gradient vanishing within a single layer, but information still must be compressed into a fixed-size state that is continuously overwritten. Long-range information can be lost through the forgetting mechanism, especially when the sequence contains many irrelevant intermediate tokens.

Despite these limitations, stacked RNNs represented a major step forward and enabled several important advances:

• Neural machine translation: The 2014 seq2seq paper with 4-layer stacked LSTMs demonstrated that end-to-end neural MT could compete with statistical MT systems, initiating the current era of NMT.
• Speech recognition: Deep stacked bidirectional LSTMs became the standard architecture for acoustic modeling, significantly reducing word error rates.
• Language modeling: Large stacked LSTMs with dropout (such as the AWD-LSTM model by Merity et al.) achieved state-of-the-art perplexity on PTB and WikiText-2, establishing benchmarks that transformers had to beat.
• Contextualized embeddings: ELMo showed that stacked bidirectional LSTMs could produce rich contextual representations that dramatically improved a wide range of NLP tasks, previewing the importance of pretraining that BERT would later exploit with transformers.

The techniques developed for stacked RNNs (variational dropout, residual connections, weight tying, and layer normalization) were not discarded when transformers arrived. They were adapted and carried forward.