Stacked RNNs › Depth vs. Width Tradeoffs

Given a fixed parameter budget, you face a choice: use a single wide layer with hidden size $d$, or use multiple narrower layers with hidden size $d' < d$. This depth-vs-width tradeoff is one of the fundamental design questions in deep learning, and recurrent networks are no exception.

For a single LSTM layer with hidden size $d$ and input size $m$, the parameter count is approximately:

$$P_{\text{single}} = 4 \times (d^2 + d \cdot m + d)$$

The factor of 4 comes from the four gates. For a 2-layer stacked LSTM where the second layer receives the first layer's hidden state as input:

$$P_{\text{stacked}} = 4(d'^2 + d' \cdot m + d') + 4(d'^2 + d' \cdot d' + d') = 4(d' \cdot m + 2d'^2 + 2d')$$

where $d'$ is the hidden size used in each layer. If we halve the hidden size ($d' = d/2$) to keep the parameter count similar, the stacked model has roughly equal parameters but much less capacity because $d'^2 = d^2/4$, which means the recurrent connections shrink quadratically.

Width gives each timestep's hidden state a larger representational capacity. A wider hidden state can store more information about the current input and the recent history. Width helps when the bottleneck is insufficient local representational capacity.

Depth adds the ability to compose representations across levels. A deep network can learn features that are functions of other features in a way that a wide network cannot easily replicate. Depth helps when the bottleneck is insufficient compositional structure.

For sequence tasks:

• Shallow but wide models (1 layer, large $d$) tend to work well on classification tasks where you need rich representations of individual timesteps but not compositional hierarchies.
• Deep but narrower models (2–4 layers, moderate $d$) tend to work better on generation and transduction tasks where multi-level structure matters.
• For equal parameter counts, depth generally wins on tasks with strong compositional structure, but width wins on tasks with limited data.