Stacked RNNs › Summary

Stacked RNNs extend single-layer recurrent networks by connecting multiple layers vertically, with each layer's output sequence serving as the next layer's input sequence. The key ideas are:

• Hierarchical temporal representations: Lower layers capture local, short-range patterns while higher layers capture long-range and semantic structure. This mirrors the compositional structure of language.
• Layer-by-layer computation: Hidden states from layer $\ell-1$ feed as inputs to layer $\ell$. Each layer maintains its own weights and state, and the entire lower layer must complete before the upper layer can run.
• Number of layers is a hyperparameter: Two layers covers most practical use cases. Three to four layers helps for complex tasks. Beyond four layers requires residual connections. The optimal depth depends on task complexity, sequence length, and dataset size.
• Depth vs. width tradeoff: For a fixed parameter budget, shallow-and-wide models work better for simple representations, while deep-and-narrow models work better for tasks with strong compositional structure.
• Dropout between layers: Standard dropout applies a different mask at each timestep (disrupting recurrent memory). Variational dropout fixes the same mask across all timesteps in a forward pass, preserving gradient flow while regularizing.
• ELMo as a stacked biLSTM: ELMo's task-specific weighting of all layers showed that different layers specialize in different linguistic properties, and that combining layers outperforms using only the final layer.
• Computational scaling: Training time and memory scale linearly with depth, imposing practical limits even before gradient flow considerations.
• Residual connections raise the depth ceiling: Without residual connections, 3 to 4 layers is the practical limit. With residual connections, 6 to 8 layers becomes feasible.

In the next part of this book, we'll explore sequence-to-sequence architectures that put stacked RNNs to work in the encoder-decoder framework for machine translation and other tasks, building directly on the concepts from this chapter.