Stacked RNNs › Stacked RNNs

In the previous chapters, we explored the building blocks of recurrent neural networks: vanilla RNNs with their vanishing gradient problems, LSTMs that solved long-range dependency issues through gating mechanisms, GRUs as a streamlined alternative, and bidirectional RNNs that read sequences in both directions. All of these operate with a single recurrent layer processing the input sequence. But some tasks require more than one level of temporal abstraction. Stacked RNNs, also called deep RNNs, address this by connecting multiple recurrent layers vertically so that each layer builds on the representations learned by the layer below it.

The core idea is simple: instead of one recurrent layer reading the raw input sequence and producing outputs, you use several layers in sequence. The output sequence of the first layer becomes the input sequence of the second, the output of the second feeds the third, and so on. This vertical stacking creates a hierarchy of temporal representations, where each layer operates on progressively more abstract summaries of the original sequence.