RNN Architecture › Stacked RNNs

Just as feedforward networks gain representational power by stacking multiple layers, RNNs can be stacked by feeding the output hidden states of one RNN layer as the input to the next.

In a stacked (deep) RNN with $L$ layers, the hidden state at layer $\ell$ is computed as:

$$\mathbf{h}_t^{(\ell)} = \tanh\!\left(\mathbf{W}_{hh}^{(\ell)}\,\mathbf{h}_{t-1}^{(\ell)} + \mathbf{W}_{xh}^{(\ell)}\,\mathbf{h}_t^{(\ell-1)} + \mathbf{b}_h^{(\ell)}\right)$$

where $\mathbf{h}_t^{(0)} = \mathbf{x}_t$ (the input) and $\mathbf{h}_t^{(\ell-1)}$ is the hidden state from the layer below at the same time step. Each layer has its own independent set of parameters.

The motivation for stacking is hierarchy. The lower layers tend to learn local, surface-level patterns (bigrams, punctuation boundaries, character-level morphology). The higher layers can then learn more abstract, long-range patterns (topic, discourse structure, semantic relationships) on top of these local summaries. This hierarchy of representations mirrors how deep feedforward networks learn a hierarchy of features in computer vision.

In practice, 2-layer stacked RNNs are common. 3 or 4 layers are used in more demanding tasks like machine translation. Beyond 4 layers, training becomes difficult due to compounding gradient instability across both depth and time. Techniques like layer normalization (applied within each RNN step) can help stabilize deeper stacked RNNs.

PyTorch's nn.RNN(num_layers=2) implements a 2-layer stacked RNN. The output tensor has shape (batch, seq_len, hidden_size) (from the final layer), and the h_n output has shape (num_layers, batch, hidden_size) containing the final hidden state of each layer.

Dropout in recurrent networks requires care. Applying standard dropout to the hidden state $\mathbf{h}_t$ at every time step disrupts the information flow through time, which can hurt performance. The standard approach (Gal & Ghahramani, 2016) is variational dropout: use the same dropout mask for all time steps within a sequence, applied only to non-recurrent connections. This drops out connections between layers (from $\mathbf{h}_t^{(\ell-1)}$ to $\mathbf{h}_t^{(\ell)}$) but not the recurrent connections ($\mathbf{h}_{t-1}^{(\ell)}$ to $\mathbf{h}_t^{(\ell)}$). PyTorch implements this through nn.RNN(dropout=0.5, num_layers=2), which applies dropout between layers but not within the recurrent step.