Stacked RNNs and Hierarchical Modeling

In the previous chapters, we explored the building blocks of recurrent neural networks: vanilla RNNs with their vanishing gradient problems, LSTMs that solved long-range dependency issues through gating mechanisms, GRUs as a streamlined alternative, and bidirectional RNNs that read sequences in both directions. All of these operate with a single recurrent layer processing the input sequence. But some tasks require more than one level of temporal abstraction. Stacked RNNs, also called deep RNNs, address this by connecting multiple recurrent layers vertically so that each layer builds on the representations learned by the layer below it.

The core idea is simple: instead of one recurrent layer reading the raw input sequence and producing outputs, you use several layers in sequence. The output sequence of the first layer becomes the input sequence of the second, the output of the second feeds the third, and so on. This vertical stacking creates a hierarchy of temporal representations, where each layer operates on progressively more abstract summaries of the original sequence.

The intuition behind stacking recurrent layers comes from an analogy with deep feedforward networks. A multilayer perceptron with a single hidden layer can approximate any continuous function, but in practice, using multiple thinner layers often learns better and requires fewer parameters than one very wide layer. The same principle applies to sequence models: multiple stacked recurrent layers can capture structure at different timescales and abstraction levels.

Think about modeling language. At the lowest level, a recurrent layer sees a sequence of word embeddings one token at a time. It builds up a local representation: which words cluster together, what short-range syntactic patterns appear, how adjacent tokens relate. The hidden state at each timestep encodes something like a compressed summary of the most recent local context.

A second layer sitting above the first doesn't see raw word embeddings at all. It sees the output sequence produced by the first layer, which is already a learned abstraction over local patterns. The second layer can now operate at a coarser timescale, learning to recognize longer-range dependencies that span what the lower layer compressed. A third layer above that operates on an even higher-level representation.

This hierarchy maps naturally onto the structure of language itself. Words compose into phrases. Phrases compose into clauses. Clauses compose into complete thoughts. No single level of processing is sufficient to capture all these scales simultaneously. Stacking layers lets the network learn each compositional level separately.

The same intuition applies to speech recognition, where phonemes compose into syllables into words into sentences, and to music modeling, where notes compose into motifs into phrases into sections. The underlying compositional structure of the data naturally benefits from a network architecture that mirrors that composition.

In a single-layer RNN, the recurrent equation takes the previous hidden state $h_{t-1}$ and the current input $x_t$ and produces the new hidden state $h_t$:

where:

• $h_t \in \mathbb{R}^d$ is the hidden state at timestep $t$
• $W_h \in \mathbb{R}^{d \times d}$ is the recurrent weight matrix
• $W_x \in \mathbb{R}^{d \times m}$ is the input projection matrix
• $x_t \in \mathbb{R}^m$ is the input at timestep $t$
• $f$ is the activation function (tanh for vanilla RNNs, sigmoid-tanh combinations for LSTMs)

In a stacked RNN with $L$ layers, the same recurrence applies, but each layer $\ell$ takes the output of the layer below as its input:

where:

• $h_t^{(\ell)} \in \mathbb{R}^{d_\ell}$ is the hidden state of layer $\ell$ at timestep $t$
• $h_t^{(\ell-1)}$ is the hidden state of the layer below, which is the "input" to layer $\ell$
• $W_h^{(\ell)}$ and $W_x^{(\ell)}$ are the recurrent and input weight matrices for layer $\ell$, each with their own learned parameters

The first layer ($\ell = 1$) receives the actual input: $h_t^{(0)} = x_t$. Each subsequent layer receives the full hidden-state sequence produced by the layer below.

The entire sequence of hidden states from layer $\ell-1$ must be computed before layer $\ell$ can run. This introduces a sequential dependency between layers: you cannot start computing layer 2 until all timesteps of layer 1 are finished. This is different from parallelism across timesteps within a single layer, which is also sequential.

For LSTMs, each layer maintains its own cell state $c_t^{(\ell)}$ in addition to its hidden state $h_t^{(\ell)}$. The LSTM equations simply replace $x_t$ with $h_t^{(\ell-1)}$:

where:

• $f_t^{(\ell)}, i_t^{(\ell)}, o_t^{(\ell)}$ are the forget, input, and output gates for layer $\ell$
• $c_t^{(\ell)}$ is the cell state for layer $\ell$
• $\tilde{c}_t^{(\ell)}$ is the candidate cell update
• $[h_{t-1}^{(\ell)},\, h_t^{(\ell-1)}]$ denotes concatenation of the layer's own previous hidden state with the current output from the layer below

Each layer in a stacked LSTM has its own complete set of gate weights and its own cell state, so the number of parameters scales linearly with the number of layers.

To make this concrete, consider a 2-layer stacked LSTM processing the sentence "the cat sat." with a hidden size of $d = 3$ (tiny, for illustration). The network processes three tokens: "the", "cat", "sat."

At timestep $t=1$ (token: "the"):

1. Layer 1 receives $x_1$ (the embedding for "the"), combines it with its initial hidden state $h_0^{(1)} = \mathbf{0}$, and computes $h_1^{(1)}$. This hidden state encodes something about "the" in the context of nothing yet seen.
2. Layer 2 receives $h_1^{(1)}$ as its input, combines it with its initial hidden state $h_0^{(2)} = \mathbf{0}$, and computes $h_1^{(2)}$. At this first timestep, the difference between layers is minimal: layer 2 simply applies another nonlinear transformation to layer 1's output.

At timestep $t=2$ (token: "cat"):

1. Layer 1 receives $x_2$ (embedding for "cat") and the previous hidden state $h_1^{(1)}$. It computes $h_2^{(1)}$, which now encodes "cat" given what came before. The recurrent state carries the context of "the" forward.
2. Layer 2 receives $h_2^{(1)}$ (layer 1's new state) and its own previous state $h_1^{(2)}$. It computes $h_2^{(2)}$, which encodes something about the relationship between layer 1's representation of "cat" and layer 2's memory of what layer 1 produced for "the".

At timestep $t=3$ (token: "sat"):

1. Layer 1 produces $h_3^{(1)}$, encoding "sat" in the context of the whole prefix "the cat."
2. Layer 2 produces $h_3^{(2)}$. By now, layer 2 has had three timesteps to build a representation that is a function of the entire sequence of layer 1 outputs. Its state represents how the sequence of local patterns from layer 1 evolved over all three tokens, rather than a direct transformation of "sat" alone.

This stepped example reveals the key asymmetry: layer 1 operates on raw inputs, and its hidden state is shaped by raw token sequences. Layer 2 operates on transformed inputs, and its hidden state is shaped by the temporal evolution of learned representations. Each timestep of layer 2 sees a more refined signal than layer 1 ever received.

The number of stacked layers is one of the primary architectural hyperparameters you control when designing a recurrent model. Unlike width (the hidden state size $d$), which affects the representational capacity of each individual timestep's encoding, depth (the number of layers) affects how many levels of abstraction the model can learn.

In practice, most successful architectures use between 2 and 4 layers:

• 1 layer: The baseline. Works well for simple sequence tasks, short sequences, and small datasets. Low computational cost.
• 2 layers: The most common choice. Adds significant representational capacity with modest additional cost. Often the best tradeoff for medium-complexity tasks.
• 3 to 4 layers: Used for demanding tasks like machine translation, speech recognition, and language modeling. Requires careful regularization.
• 5+ layers: Rarely beneficial without residual connections. Gradient flow becomes difficult, and training becomes unstable.

The right number of layers depends on several factors:

• Task complexity: More compositional structure in the target requires more layers. Sentence-level classification might need 2, document-level summarization might need 3 to 4.
• Sequence length: Longer sequences have more temporal structure, so more layers can help. Short sequences (length < 20) rarely benefit from more than 2 layers.
• Dataset size: More layers mean more parameters. With small datasets, deep stacked RNNs overfit. Use fewer layers and stronger regularization.
• Computational budget: Each additional layer roughly doubles training time and memory usage (for a fixed hidden size).

A common search strategy is to start with 1 layer, establish a baseline, add 1 layer at a time, and stop when validation performance stops improving or overfitting becomes severe.

Given a fixed parameter budget, you face a choice: use a single wide layer with hidden size $d$, or use multiple narrower layers with hidden size $d' < d$. This depth-vs-width tradeoff is one of the fundamental design questions in deep learning, and recurrent networks are no exception.

For a single LSTM layer with hidden size $d$ and input size $m$, the parameter count is approximately:

The factor of 4 comes from the four gates. For a 2-layer stacked LSTM where the second layer receives the first layer's hidden state as input:

where $d'$ is the hidden size used in each layer. If we halve the hidden size ($d' = d/2$) to keep the parameter count similar, the stacked model has roughly equal parameters but much less capacity because $d'^2 = d^2/4$, which means the recurrent connections shrink quadratically.

Width gives each timestep's hidden state a larger representational capacity. A wider hidden state can store more information about the current input and the recent history. Width helps when the bottleneck is insufficient local representational capacity.

Depth adds the ability to compose representations across levels. A deep network can learn features that are functions of other features in a way that a wide network cannot easily replicate. Depth helps when the bottleneck is insufficient compositional structure.

For sequence tasks:

• Shallow but wide models (1 layer, large $d$) tend to work well on classification tasks where you need rich representations of individual timesteps but not compositional hierarchies.
• Deep but narrower models (2–4 layers, moderate $d$) tend to work better on generation and transduction tasks where multi-level structure matters.
• For equal parameter counts, depth generally wins on tasks with strong compositional structure, but width wins on tasks with limited data.

The plot illustrates a consistent pattern: at equal parameter budgets, 2 to 3 layers with a moderate hidden size outperform a single wide layer for language modeling, but going beyond 3 to 4 layers without residual connections begins to hurt performance.

Deeper models are more expressive but also more prone to overfitting. The standard solution in feedforward networks is dropout, which randomly zeroes out a fraction $p$ of activations during training. Applying dropout naively to recurrent networks requires more care.

The original approach, used in early stacked RNNs, applies standard dropout only between layers (on the vertical connections), not on the recurrent connections within a layer. Concretely, the input to layer $\ell$ at timestep $t$ is:

where $\text{Dropout}(x, p)$ zeroes each element of $x$ independently with probability $p$ and scales by $\frac{1}{1-p}$ to preserve expected values.

The key constraint is that dropout is applied independently at each timestep. This means the dropout mask changes at every timestep. Applying a different mask at each timestep destroys the ability of the network to retain information over time, because any bit of memory that gets dropped at one step is lost permanently for the recurrent path through that step.

Gal and Ghahramani (2016) showed that the correct Bayesian interpretation of dropout for recurrent networks requires using the same dropout mask at every timestep within a single forward pass. This is called variational dropout.

Instead of sampling a new mask at every $t$, you sample once per forward pass:

and apply these fixed masks consistently:

where $\odot$ is element-wise multiplication. Using a fixed mask across timesteps means the same units are always dropped within a sequence, which preserves gradient flow through time while still regularizing.

The practical effect is significant. With standard timestep-varying dropout, a model trained with high dropout rates struggles to retain information across many steps because the dropout noise accumulates over the recurrence. With variational dropout, a consistent mask prevents specific units from participating, but the units that do participate are always the same, so gradients flow cleanly through those units across all timesteps.

In PyTorch's built-in nn.LSTM with dropout=p, the dropout is applied between layers (inter-layer) using standard dropout. For variational dropout, you need a custom implementation or a library like torchnlp or WeightDrop from Merity et al.

Typical dropout rates for stacked RNNs:

• 0.1–0.2: Light regularization, suitable for large datasets
• 0.3–0.5: Standard regularization, good for medium datasets
• 0.5: Strong regularization, often used for small datasets or highly expressive models

The optimal dropout rate usually increases with depth and decreases with dataset size. For a 3-layer stacked LSTM on a medium-sized language modeling task, rates of 0.3–0.5 between layers are common.

The divergence between training and validation loss without dropout illustrates classic overfitting. Standard dropout closes the gap somewhat, while variational dropout achieves the lowest validation loss because it preserves coherent gradient flow while still regularizing.

Machine translation was one of the first real-world tasks where stacked LSTMs demonstrated dramatic improvements over single-layer models. The landmark result was Sutskever, Vinyals, and Le's 2014 paper "Sequence to Sequence Learning with Neural Networks," which used a 4-layer stacked LSTM encoder-decoder architecture and achieved state-of-the-art results on English-to-French translation.

The seq2seq architecture for machine translation consists of:

1. Encoder: A stacked LSTM reads the source sentence and compresses it into a fixed-size context vector (the final hidden state of the top layer).
2. Decoder: A separate stacked LSTM takes the context vector as its initial hidden state and generates the target sentence one token at a time.

The original model used 4 layers of 1000-unit LSTMs in both the encoder and decoder. The input sentence was fed in reverse order to the encoder, a trick that brought the beginning of the source sentence closer to the beginning of the target sentence in the unrolled computation graph, making gradient flow easier.

The hidden state tensor has shape (num_layers, batch, hidden_size), meaning each layer maintains its own separate hidden state. When the encoder finishes reading the source sentence, you pass all layers' hidden and cell states to the decoder, initializing its entire stack with the compressed source representation.

The Sutskever et al. paper explicitly tested 1, 2, 4, and 8 layer models. Their key finding: 4 layers outperformed 1 and 2 layers substantially, while 8 layers degraded (without residual connections). The improvement from 4 layers came from the encoder learning a richer, more hierarchical representation of the source sentence. The final context vector captures both local word-level information (from lower layers) and global sentence structure (from higher layers).

Let's look more closely at implementing variational dropout, which is the recommended approach for regularizing stacked RNNs. The key requirement is that the same mask is used across all timesteps in a single forward pass.

The within-pass variation is near zero because the same mask applies at every timestep, confirming that information can flow consistently through the recurrence. The cross-pass variation shows that different forward passes do get different masks. This provides the stochastic regularization.

One of the most influential applications of stacked RNNs is ELMo (Embeddings from Language Models), introduced by Peters et al. in 2018. ELMo uses a 2-layer stacked bidirectional LSTM trained as a language model, and the resulting representations became one of the first major successes of contextualized word embeddings before the transformer era.

ELMo combines several ideas discussed in earlier chapters:

1. Bidirectional RNN (see Chapter 8 of this part): Two separate LSTMs, one reading forward and one reading backward
2. Stacking: 2 layers of bidirectional LSTMs, where each direction is stacked independently
3. Layer combination: ELMo doesn't use just the top layer. It uses a learned weighted combination of all layers

The architecture processes each sentence through:

• A character-level CNN to produce token embeddings (avoiding the fixed vocabulary problem)
• A forward 2-layer stacked LSTM reading left to right
• A backward 2-layer stacked LSTM reading right to left
• Concatenation of forward and backward states at each layer

For a token $t$ in a sequence of length $T$, ELMo produces representations at three levels:

• $E_0(t)$: The character-CNN embedding (context-independent)
• $E_1(t)$: The concatenated forward/backward states from layer 1 (local context)
• $E_2(t)$: The concatenated forward/backward states from layer 2 (global context)

The final ELMo embedding for downstream tasks is:

where:

• $s_\ell$ are task-specific softmax weights (learned scalars that sum to 1)
• $\gamma$ is a task-specific scalar that scales the overall magnitude
• The sum runs over all three levels: character embedding + 2 LSTM layers

The key insight is that different layers capture different types of linguistic information:

• Layer 1 captures syntax: part-of-speech, dependency structure
• Layer 2 captures semantics: word sense disambiguation, coreference

By learning to combine layers, different downstream tasks can emphasize the information that matters most. A POS tagger uses mostly layer 1. A coreference model uses mostly layer 2. The task itself determines which level of the hierarchy is most relevant.

Before ELMo, most NLP systems used static word embeddings like Word2Vec or GloVe, which give every token the same representation regardless of context. ELMo demonstrated that contextual representations from stacked recurrent networks could dramatically improve downstream tasks. Improvements of 4–14% absolute on tasks like question answering, textual entailment, and coreference resolution were reported at the time of publication.

ELMo also highlighted an important property of stacked RNNs that pure depth metrics miss: different layers specialize. The lower layers specialize in syntax, the upper layers in semantics. Using all layers together, rather than just the final layer, extracted the most value from the stack.

The visualization confirms the layer specialization pattern. Syntactic tasks like POS tagging heavily favor Layer 1, while semantic tasks like coreference resolution rely primarily on Layer 2. This task-adaptive weighting is what makes ELMo more powerful than simply using the final layer's representation.

One concern with stacking recurrent layers is gradient flow. In a stacked RNN, gradients must travel backward through both the depth dimension (across layers) and the time dimension (across timesteps). There are two distinct paths for gradient decay:

Across time (within a layer): For a vanilla RNN, this is the vanishing gradient problem covered in the Vanishing Gradients chapter. LSTMs solve this within a single layer via the cell state highway.

Across layers (at a single timestep): Even with LSTMs, adding more layers creates a new source of gradient attenuation. Each layer applies a tanh activation and a gating operation, and gradients multiplied through many layers can still shrink.

For a 4-layer stacked LSTM, the gradient for a parameter in layer 1 must pass through 3 additional layers before reaching the output. If each layer gate reduces the gradient by a factor of $\gamma < 1$, the effective gradient at layer 1 is $\gamma^3$ times what it would be at layer 4. For deep stacks, this vertical gradient attenuation becomes the bottleneck.

The standard solution in modern deep stacked RNNs is residual connections, also called skip connections. Originally developed for convolutional networks (ResNet), residual connections add the layer's input directly to its output:

where $f(\cdot)$ represents the full LSTM computation for layer $\ell$. The additive shortcut provides a direct gradient path: the derivative of the loss with respect to $h_t^{(\ell-1)}$ includes both the gradient through $f$ and a direct gradient of 1.0. This prevents complete vanishing.

For residual connections to work, the output dimension of $f$ must match the input dimension. If layers have different sizes, a linear projection is added:

Residual connections allowed much deeper stacking. The original Google Neural Machine Translation (GNMT) system from 2016 used 8 stacked LSTMs with residual connections in both the encoder and decoder.

The gradient difference between the output layer (Layer 4) and the deepest layer (Layer 1) is much more pronounced without residual connections. With residual connections, the lower layers still receive meaningful gradient signal, allowing the full depth of the network to participate in learning.

Stacking layers has a direct and significant impact on computational cost. Understanding these costs helps you make informed decisions about architecture design.

For a single LSTM layer processing a sequence of length $T$ with hidden size $d$, the computation at each timestep involves matrix multiplications of size $O(d^2)$ (the recurrent matrix $W_h$) and $O(d \cdot m)$ (the input matrix $W_x$). The total time for one forward pass is:

For a stacked model with $L$ layers and the same hidden size $d$ (and noting that after the first layer the "input" is also size $d$):

Time scales linearly with the number of layers. A 4-layer model takes approximately 4 times as long per batch as a 1-layer model with the same hidden size.

During training, you must store intermediate activations for backpropagation. For a single-layer LSTM:

• Hidden states: $O(T \cdot d)$ per sample
• Cell states: $O(T \cdot d)$ per sample
• Gate activations: $O(T \cdot d)$ per sample (4 gates)

Total memory per sample: $O(T \cdot d)$

For an $L$-layer stack, memory scales as $O(T \cdot L \cdot d)$: you must store the full state history for every layer to compute gradients.

For a 4-layer LSTM processing sequences of length 100 with hidden size 512:

• Memory for activations: $100 \times 4 \times 512 \times 4 \approx 800$K floats per sample
• At batch size 32: roughly 100 MB just for LSTM activations (in float32)

This is why long sequences with deep stacks require careful batch size management.

Parameter count for a stacked LSTM with $L$ layers, input size $m$, and hidden size $d$:

Simplified: the first layer has $O(d \cdot m + d^2)$ parameters and each subsequent layer has $O(d^2)$ parameters. Total parameters scale as $O(L \cdot d^2)$ for $L \gg 1$.

The linear scaling in both time and memory confirms that adding layers has predictable costs. For a given computational budget, this creates a hard cap on useful depth: a 5-layer model that takes 5x the training time and fills all available GPU memory might not be the right choice even if, in theory, it could learn better representations.

PyTorch's nn.LSTM supports stacking natively through the num_layers parameter. Let's walk through a complete implementation covering the key design decisions.

The key parameters for stacked LSTM architectures are:

• num_layers: The depth of the stack. Start at 2 for most tasks. Add layers only if validation performance keeps improving. Values of 2–4 cover the vast majority of practical use cases.
• hidden_size: The width of each layer. This is the dominant factor in parameter count because $d^2$ grows quadratically. Common values: 256 (small), 512 (medium), 1024 (large).
• dropout: The inter-layer dropout rate. Applied between all consecutive layer pairs when num_layers > 1. Typical values: 0.3–0.5.
• embed_dim: The dimensionality of the token embeddings fed to the first layer. Often set equal to hidden_size to avoid an asymmetric first layer.

In practice, stacking beyond 4–6 layers of plain RNNs (without residual connections or other architectural support) rarely helps and often hurts. The reasons span gradient dynamics, optimization difficulty, and computational cost:

Gradient attenuation across layers: As discussed, gradients weaken as they flow through multiple layers. Without residual connections, gradients in layer 1 receive a fraction of the signal available at layer 4. The lower layers effectively stop learning, wasting their capacity.

Optimization landscape difficulty: Deeper networks have more complex loss surfaces with more saddle points and potential for poor local minima. The combination of depth (across layers) and temporal recurrence (across time) creates particularly challenging optimization problems. Gradient clipping (covered in a previous chapter) becomes essential.

Diminishing returns: In empirical studies, the improvement from adding each additional layer decreases. Going from 1 to 2 layers often yields a significant improvement. Going from 3 to 4 layers yields a smaller one. Going from 4 to 5 layers may show no improvement or degradation.

Memory and time cost: As shown in the scaling analysis, each layer adds proportional cost. At some point, spending the same computational budget on a wider or better-regularized shallow model outperforms the deeper alternative.

With residual connections, the practical depth limit extends to 8 to 12 layers. The Google Neural Machine Translation (GNMT) system used 8-layer stacked LSTMs. The RWTH Aachen speech recognition system achieved excellent results with 6 to 8 layers. In both cases, residual connections were needed.

Another technique that helps with depth is layer normalization, which normalizes the activations across features (not across a batch) at each timestep. For a hidden state $h_t^{(\ell)}$:

where:

• $\mu_t^{(\ell)}$ is the mean of the hidden state across features
• $\sigma_t^{(\ell)}$ is the standard deviation across features
• $\gamma$ and $\beta$ are learned scale and shift parameters
• $\epsilon$ is a small constant for numerical stability

Layer normalization stabilizes the distribution of activations entering each layer, reducing the sensitivity to initialization and allowing higher learning rates. It works particularly well for RNNs because, unlike batch normalization, it doesn't depend on batch statistics and can be applied without modification at inference time.

The diverging curves after each architecture's optimal depth illustrate the two distinct failure modes: plain stacks degrade early due to gradient attenuation, while residual stacks degrade later and more gradually as optimization difficulty and memory constraints become the limiting factors.

Stacked RNNs achieved remarkable results in their time, but they come with significant limitations that motivated the eventual shift to transformers.

The fundamental constraint of any RNN, stacked or not, is that computation is sequential in time. To compute the hidden state at timestep $t$, you need the hidden state at timestep $t-1$. This means you cannot parallelize across the time dimension. For a sequence of length $T$ with $L$ layers, the critical path has length $T \times L$.

Modern accelerators (GPUs and TPUs) achieve peak performance through massive parallelism. Sequential computation underutilizes them. A transformer, by contrast, processes all positions in parallel using attention, allowing it to better exploit modern hardware. This parallelism advantage translates into faster training at the same parameter count.

For a sequence of length 512, a stacked 4-layer LSTM has a sequential critical path of 2048 steps. A transformer with the same depth (4 layers) processes all 512 positions simultaneously, with a critical path of just 4 steps (one per layer). This is a 128x difference in parallelism.

Even with LSTM gating and stacking, learning dependencies that span hundreds or thousands of tokens is challenging. The LSTM cell state does mitigate gradient vanishing within a single layer, but information still must be compressed into a fixed-size state that is continuously overwritten. Long-range information can be lost through the forgetting mechanism, especially when the sequence contains many irrelevant intermediate tokens.

Despite these limitations, stacked RNNs represented a major step forward and enabled several important advances:

• Neural machine translation: The 2014 seq2seq paper with 4-layer stacked LSTMs demonstrated that end-to-end neural MT could compete with statistical MT systems, initiating the current era of NMT.
• Speech recognition: Deep stacked bidirectional LSTMs became the standard architecture for acoustic modeling, significantly reducing word error rates.
• Language modeling: Large stacked LSTMs with dropout (such as the AWD-LSTM model by Merity et al.) achieved state-of-the-art perplexity on PTB and WikiText-2, establishing benchmarks that transformers had to beat.
• Contextualized embeddings: ELMo showed that stacked bidirectional LSTMs could produce rich contextual representations that dramatically improved a wide range of NLP tasks, previewing the importance of pretraining that BERT would later exploit with transformers.

The techniques developed for stacked RNNs (variational dropout, residual connections, weight tying, and layer normalization) were not discarded when transformers arrived. They were adapted and carried forward.

Stacked RNNs extend single-layer recurrent networks by connecting multiple layers vertically, with each layer's output sequence serving as the next layer's input sequence. The key ideas are:

• Hierarchical temporal representations: Lower layers capture local, short-range patterns while higher layers capture long-range and semantic structure. This mirrors the compositional structure of language.
• Layer-by-layer computation: Hidden states from layer $\ell-1$ feed as inputs to layer $\ell$. Each layer maintains its own weights and state, and the entire lower layer must complete before the upper layer can run.
• Number of layers is a hyperparameter: Two layers covers most practical use cases. Three to four layers helps for complex tasks. Beyond four layers requires residual connections. The optimal depth depends on task complexity, sequence length, and dataset size.
• Depth vs. width tradeoff: For a fixed parameter budget, shallow-and-wide models work better for simple representations, while deep-and-narrow models work better for tasks with strong compositional structure.
• Dropout between layers: Standard dropout applies a different mask at each timestep (disrupting recurrent memory). Variational dropout fixes the same mask across all timesteps in a forward pass, preserving gradient flow while regularizing.
• ELMo as a stacked biLSTM: ELMo's task-specific weighting of all layers showed that different layers specialize in different linguistic properties, and that combining layers outperforms using only the final layer.
• Computational scaling: Training time and memory scale linearly with depth, imposing practical limits even before gradient flow considerations.
• Residual connections raise the depth ceiling: Without residual connections, 3 to 4 layers is the practical limit. With residual connections, 6 to 8 layers becomes feasible.

In the next part of this book, we'll explore sequence-to-sequence architectures that put stacked RNNs to work in the encoder-decoder framework for machine translation and other tasks, building directly on the concepts from this chapter.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about stacked RNNs and hierarchical sequence modeling.