Stacked RNNs › Dropout Between RNN Layers

Deeper models are more expressive but also more prone to overfitting. The standard solution in feedforward networks is dropout, which randomly zeroes out a fraction $p$ of activations during training. Applying dropout naively to recurrent networks requires more care.

The original approach, used in early stacked RNNs, applies standard dropout only between layers (on the vertical connections), not on the recurrent connections within a layer. Concretely, the input to layer $\ell$ at timestep $t$ is:

$$\tilde{h}_t^{(\ell-1)} = \text{Dropout}\!\left(h_t^{(\ell-1)},\, p\right)$$

where $\text{Dropout}(x, p)$ zeroes each element of $x$ independently with probability $p$ and scales by $\frac{1}{1-p}$ to preserve expected values.

The key constraint is that dropout is applied independently at each timestep. This means the dropout mask changes at every timestep. Applying a different mask at each timestep destroys the ability of the network to retain information over time, because any bit of memory that gets dropped at one step is lost permanently for the recurrent path through that step.

Gal and Ghahramani (2016) showed that the correct Bayesian interpretation of dropout for recurrent networks requires using the same dropout mask at every timestep within a single forward pass. This is called variational dropout.

Instead of sampling a new mask at every $t$, you sample once per forward pass:

$$\epsilon_h \sim \text{Bernoulli}(1-p), \quad \epsilon_x \sim \text{Bernoulli}(1-p)$$

and apply these fixed masks consistently:

$$\tilde{h}_{t-1}^{(\ell)} = h_{t-1}^{(\ell)} \odot \epsilon_h, \quad \tilde{h}_t^{(\ell-1)} = h_t^{(\ell-1)} \odot \epsilon_x$$

where $\odot$ is element-wise multiplication. Using a fixed mask across timesteps means the same units are always dropped within a sequence, which preserves gradient flow through time while still regularizing.

The practical effect is significant. With standard timestep-varying dropout, a model trained with high dropout rates struggles to retain information across many steps because the dropout noise accumulates over the recurrence. With variational dropout, a consistent mask prevents specific units from participating, but the units that do participate are always the same, so gradients flow cleanly through those units across all timesteps.

In PyTorch's built-in nn.LSTM with dropout=p, the dropout is applied between layers (inter-layer) using standard dropout. For variational dropout, you need a custom implementation or a library like torchnlp or WeightDrop from Merity et al.

Typical dropout rates for stacked RNNs:

• 0.1–0.2: Light regularization, suitable for large datasets
• 0.3–0.5: Standard regularization, good for medium datasets
• 0.5: Strong regularization, often used for small datasets or highly expressive models

The optimal dropout rate usually increases with depth and decreases with dataset size. For a 3-layer stacked LSTM on a medium-sized language modeling task, rates of 0.3–0.5 between layers are common.