Stacked RNNs › Gradient Flow in Deep RNNs

One concern with stacking recurrent layers is gradient flow. In a stacked RNN, gradients must travel backward through both the depth dimension (across layers) and the time dimension (across timesteps). There are two distinct paths for gradient decay:

Across time (within a layer): For a vanilla RNN, this is the vanishing gradient problem covered in the Vanishing Gradients chapter. LSTMs solve this within a single layer via the cell state highway.

Across layers (at a single timestep): Even with LSTMs, adding more layers creates a new source of gradient attenuation. Each layer applies a tanh activation and a gating operation, and gradients multiplied through many layers can still shrink.

For a 4-layer stacked LSTM, the gradient for a parameter in layer 1 must pass through 3 additional layers before reaching the output. If each layer gate reduces the gradient by a factor of $\gamma < 1$, the effective gradient at layer 1 is $\gamma^3$ times what it would be at layer 4. For deep stacks, this vertical gradient attenuation becomes the bottleneck.

The standard solution in modern deep stacked RNNs is residual connections, also called skip connections. Originally developed for convolutional networks (ResNet), residual connections add the layer's input directly to its output:

$$h_t^{(\ell)} = f\!\left(h_t^{(\ell-1)}\right) + h_t^{(\ell-1)}$$

where $f(\cdot)$ represents the full LSTM computation for layer $\ell$. The additive shortcut provides a direct gradient path: the derivative of the loss with respect to $h_t^{(\ell-1)}$ includes both the gradient through $f$ and a direct gradient of 1.0. This prevents complete vanishing.

For residual connections to work, the output dimension of $f$ must match the input dimension. If layers have different sizes, a linear projection is added:

$$h_t^{(\ell)} = f\!\left(h_t^{(\ell-1)}\right) + W_{\text{proj}}\, h_t^{(\ell-1)}$$

Residual connections allowed much deeper stacking. The original Google Neural Machine Translation (GNMT) system from 2016 used 8 stacked LSTMs with residual connections in both the encoder and decoder.