Bidirectional RNNs › Summary

Bidirectional RNNs extend recurrent architectures to process sequences in both directions simultaneously. The key ideas are:

• Two passes: A forward network processes left to right; a backward network processes right to left. Both networks process the same input but have independent weight matrices.
• Concatenation: At each position, the forward and backward hidden states are concatenated, producing a representation that captures both left and right context.
• Full-context encoding: Every token's representation summarizes the entire surrounding sequence. This is the critical advantage over unidirectional RNNs for encoding tasks.
• Encoder-decoder split: Bidirectionality belongs in encoders, where the full input is available. Decoders must remain unidirectional to support autoregressive generation.
• Sequence labeling applications: BiRNNs excel at NER, POS tagging, chunking, semantic role labeling, and any task where per-token predictions depend on both local and global context. BiLSTM-CRF became the standard NER architecture before BERT.
• Sentence-level pooling: For tasks requiring a single sentence vector, strategies include concatenating final hidden states, mean pooling, max pooling, or attention-weighted pooling. Each makes different trade-offs between simplicity and expressiveness.
• Stacking and depth: Multiple stacked bidirectional layers build hierarchical representations, with lower layers capturing syntax and upper layers capturing semantics.
• ELMo and the transition to transformers: ELMo showed that pretrained deep BiLSTMs produce powerful contextual representations. BERT replaced the sequential BiLSTM with parallel self-attention, achieving joint bidirectionality rather than concatenated separate unidirectional LMs.
• Generation incompatibility: Bidirectional processing requires the full input sequence to be available in advance, making it incompatible with autoregressive generation. Language models must remain unidirectional.
• PyTorch usage: Setting bidirectional=True in nn.LSTM or nn.GRU enables bidirectionality. The output dimension becomes hidden_size * 2 because the two directional states are concatenated.

The next chapter examines stacked RNNs, where multiple recurrent layers are arranged vertically to build hierarchical sequence representations. The insights from bidirectionality carry forward: stacked bidirectional layers are a common pattern in high-performance sequence encoders.