Stacked RNNs › Number of Layers as a Hyperparameter

The number of stacked layers is one of the primary architectural hyperparameters you control when designing a recurrent model. Unlike width (the hidden state size $d$), which affects the representational capacity of each individual timestep's encoding, depth (the number of layers) affects how many levels of abstraction the model can learn.

In practice, most successful architectures use between 2 and 4 layers:

• 1 layer: The baseline. Works well for simple sequence tasks, short sequences, and small datasets. Low computational cost.
• 2 layers: The most common choice. Adds significant representational capacity with modest additional cost. Often the best tradeoff for medium-complexity tasks.
• 3 to 4 layers: Used for demanding tasks like machine translation, speech recognition, and language modeling. Requires careful regularization.
• 5+ layers: Rarely beneficial without residual connections. Gradient flow becomes difficult, and training becomes unstable.

The right number of layers depends on several factors:

• Task complexity: More compositional structure in the target requires more layers. Sentence-level classification might need 2, document-level summarization might need 3 to 4.
• Sequence length: Longer sequences have more temporal structure, so more layers can help. Short sequences (length < 20) rarely benefit from more than 2 layers.
• Dataset size: More layers mean more parameters. With small datasets, deep stacked RNNs overfit. Use fewer layers and stronger regularization.
• Computational budget: Each additional layer roughly doubles training time and memory usage (for a fixed hidden size).

A common search strategy is to start with 1 layer, establish a baseline, add 1 layer at a time, and stop when validation performance stops improving or overfitting becomes severe.