Stacked RNNs › How Hidden States Feed the Next Layer

In a single-layer RNN, the recurrent equation takes the previous hidden state $h_{t-1}$ and the current input $x_t$ and produces the new hidden state $h_t$:

$$h_t = f(W_h h_{t-1} + W_x x_t + b)$$

where:

• $h_t \in \mathbb{R}^d$ is the hidden state at timestep $t$
• $W_h \in \mathbb{R}^{d \times d}$ is the recurrent weight matrix
• $W_x \in \mathbb{R}^{d \times m}$ is the input projection matrix
• $x_t \in \mathbb{R}^m$ is the input at timestep $t$
• $f$ is the activation function (tanh for vanilla RNNs, sigmoid-tanh combinations for LSTMs)

In a stacked RNN with $L$ layers, the same recurrence applies, but each layer $\ell$ takes the output of the layer below as its input:

$$h_t^{(\ell)} = f\!\left(W_h^{(\ell)} h_{t-1}^{(\ell)} + W_x^{(\ell)} h_t^{(\ell-1)} + b^{(\ell)}\right)$$

where:

• $h_t^{(\ell)} \in \mathbb{R}^{d_\ell}$ is the hidden state of layer $\ell$ at timestep $t$
• $h_t^{(\ell-1)}$ is the hidden state of the layer below, which is the "input" to layer $\ell$
• $W_h^{(\ell)}$ and $W_x^{(\ell)}$ are the recurrent and input weight matrices for layer $\ell$, each with their own learned parameters

The first layer ($\ell = 1$) receives the actual input: $h_t^{(0)} = x_t$. Each subsequent layer receives the full hidden-state sequence produced by the layer below.

The entire sequence of hidden states from layer $\ell-1$ must be computed before layer $\ell$ can run. This introduces a sequential dependency between layers: you cannot start computing layer 2 until all timesteps of layer 1 are finished. This is different from parallelism across timesteps within a single layer, which is also sequential.

For LSTMs, each layer maintains its own cell state $c_t^{(\ell)}$ in addition to its hidden state $h_t^{(\ell)}$. The LSTM equations simply replace $x_t$ with $h_t^{(\ell-1)}$:

$$\begin{aligned} f_t^{(\ell)} &= \sigma\!\left(W_f^{(\ell)} [h_{t-1}^{(\ell)},\, h_t^{(\ell-1)}] + b_f^{(\ell)}\right) \\ i_t^{(\ell)} &= \sigma\!\left(W_i^{(\ell)} [h_{t-1}^{(\ell)},\, h_t^{(\ell-1)}] + b_i^{(\ell)}\right) \\ \tilde{c}_t^{(\ell)} &= \tanh\!\left(W_c^{(\ell)} [h_{t-1}^{(\ell)},\, h_t^{(\ell-1)}] + b_c^{(\ell)}\right) \\ c_t^{(\ell)} &= f_t^{(\ell)} \odot c_{t-1}^{(\ell)} + i_t^{(\ell)} \odot \tilde{c}_t^{(\ell)} \\ o_t^{(\ell)} &= \sigma\!\left(W_o^{(\ell)} [h_{t-1}^{(\ell)},\, h_t^{(\ell-1)}] + b_o^{(\ell)}\right) \\ h_t^{(\ell)} &= o_t^{(\ell)} \odot \tanh(c_t^{(\ell)}) \end{aligned}$$

where:

• $f_t^{(\ell)}, i_t^{(\ell)}, o_t^{(\ell)}$ are the forget, input, and output gates for layer $\ell$
• $c_t^{(\ell)}$ is the cell state for layer $\ell$
• $\tilde{c}_t^{(\ell)}$ is the candidate cell update
• $[h_{t-1}^{(\ell)},\, h_t^{(\ell-1)}]$ denotes concatenation of the layer's own previous hidden state with the current output from the layer below

Each layer in a stacked LSTM has its own complete set of gate weights and its own cell state, so the number of parameters scales linearly with the number of layers.

To make this concrete, consider a 2-layer stacked LSTM processing the sentence "the cat sat." with a hidden size of $d = 3$ (tiny, for illustration). The network processes three tokens: "the", "cat", "sat."

At timestep $t=1$ (token: "the"):

1. Layer 1 receives $x_1$ (the embedding for "the"), combines it with its initial hidden state $h_0^{(1)} = \mathbf{0}$, and computes $h_1^{(1)}$. This hidden state encodes something about "the" in the context of nothing yet seen.
2. Layer 2 receives $h_1^{(1)}$ as its input, combines it with its initial hidden state $h_0^{(2)} = \mathbf{0}$, and computes $h_1^{(2)}$. At this first timestep, the difference between layers is minimal: layer 2 simply applies another nonlinear transformation to layer 1's output.

At timestep $t=2$ (token: "cat"):

1. Layer 1 receives $x_2$ (embedding for "cat") and the previous hidden state $h_1^{(1)}$. It computes $h_2^{(1)}$, which now encodes "cat" given what came before. The recurrent state carries the context of "the" forward.
2. Layer 2 receives $h_2^{(1)}$ (layer 1's new state) and its own previous state $h_1^{(2)}$. It computes $h_2^{(2)}$, which encodes something about the relationship between layer 1's representation of "cat" and layer 2's memory of what layer 1 produced for "the".

At timestep $t=3$ (token: "sat"):

1. Layer 1 produces $h_3^{(1)}$, encoding "sat" in the context of the whole prefix "the cat."
2. Layer 2 produces $h_3^{(2)}$. By now, layer 2 has had three timesteps to build a representation that is a function of the entire sequence of layer 1 outputs. Its state represents how the sequence of local patterns from layer 1 evolved over all three tokens, rather than a direct transformation of "sat" alone.

This stepped example reveals the key asymmetry: layer 1 operates on raw inputs, and its hidden state is shaped by raw token sequences. Layer 2 operates on transformed inputs, and its hidden state is shaped by the temporal evolution of learned representations. Each timestep of layer 2 sees a more refined signal than layer 1 ever received.