Bidirectional RNNs › Stacked Bidirectional RNNs

A natural extension of the single-layer BiRNN is to stack multiple bidirectional layers vertically, creating a deep bidirectional architecture. In a stacked BiRNN with $L$ layers, the output of each layer (the concatenated forward and backward hidden states at each position) becomes the input to the next layer above it. This creates a hierarchy of representations: lower layers tend to capture fine-grained local patterns, while upper layers capture broader syntactic and semantic structure.

The mechanics of stacking are straightforward in PyTorch: setting num_layers=2 in an nn.LSTM with bidirectional=True creates two stacked BiLSTM layers. Each layer operates on the full sequence with its own independent pair of forward and backward LSTMs. The output of the first layer, with shape (batch, seq_len, 2 * hidden_dim), becomes the input to the second layer, where the 2 * hidden_dim is the input dimension. The second layer also produces outputs with shape (batch, seq_len, 2 * hidden_dim).

This means the input dimension to the second layer is twice the hidden dimension of the first layer. If the first layer has hidden size 128, its output is 256-dimensional (128 from the forward pass, 128 from the backward pass), and the second layer's LSTM cells take 256-dimensional inputs. This is correct behavior, but it is easy to accidentally create parameter mismatches if you are constructing stacked layers manually.

Each layer in a stacked BiRNN can develop qualitatively different representations. In studies of deep BiLSTMs trained for language understanding tasks, the pattern is fairly consistent: lower layers specialize in morphological and syntactic information (word types, phrase boundaries, grammatical relationships), while upper layers specialize in semantic and pragmatic information (entity types, argument roles, discourse coherence).

This hierarchy is not imposed by the architecture: it emerges from training. The reason it emerges is that semantic information often requires syntactic information as a prerequisite. To determine that "Apple" is a company rather than a fruit, you first need to know that "Apple" is a noun (syntactic), then that it is a proper noun, then that the surrounding context is corporate or technological (semantic). The lower layers of a stacked BiLSTM naturally handle the earlier steps in this reasoning chain, and the upper layers handle the later steps.

Deeper architectures are more expressive but also more prone to overfitting. Several regularization strategies are commonly applied to stacked BiRNNs:

Dropout between layers is the most common. After the output of each layer (but before feeding it into the next), a dropout mask is applied, randomly zeroing some dimensions. PyTorch's nn.LSTM supports this directly via the dropout parameter, which applies dropout to all layers except the last.

Variational dropout (also called recurrent dropout) applies the same dropout mask consistently across all time steps within a single sequence, rather than sampling a new mask at each step. Standard dropout applied at each time step disrupts the recurrent state's temporal coherence; variational dropout preserves it while still providing regularization. This approach was popularized by Gal and Ghahramani in 2016 and is commonly used in high-performance BiLSTM systems.

Layer normalization can be applied within each LSTM cell to stabilize training in very deep stacks. This is less common than dropout but useful when training very deep models or when batch statistics are unreliable (small batches or very variable sequence lengths).