Stacked RNNs › Hierarchical Temporal Representations

The intuition behind stacking recurrent layers comes from an analogy with deep feedforward networks. A multilayer perceptron with a single hidden layer can approximate any continuous function, but in practice, using multiple thinner layers often learns better and requires fewer parameters than one very wide layer. The same principle applies to sequence models: multiple stacked recurrent layers can capture structure at different timescales and abstraction levels.

Think about modeling language. At the lowest level, a recurrent layer sees a sequence of word embeddings one token at a time. It builds up a local representation: which words cluster together, what short-range syntactic patterns appear, how adjacent tokens relate. The hidden state at each timestep encodes something like a compressed summary of the most recent local context.

A second layer sitting above the first doesn't see raw word embeddings at all. It sees the output sequence produced by the first layer, which is already a learned abstraction over local patterns. The second layer can now operate at a coarser timescale, learning to recognize longer-range dependencies that span what the lower layer compressed. A third layer above that operates on an even higher-level representation.

This hierarchy maps naturally onto the structure of language itself. Words compose into phrases. Phrases compose into clauses. Clauses compose into complete thoughts. No single level of processing is sufficient to capture all these scales simultaneously. Stacking layers lets the network learn each compositional level separately.

The same intuition applies to speech recognition, where phonemes compose into syllables into words into sentences, and to music modeling, where notes compose into motifs into phrases into sections. The underlying compositional structure of the data naturally benefits from a network architecture that mirrors that composition.