RNN Architecture › Bidirectional RNNs

A standard (unidirectional) RNN processes a sequence from left to right. At each position $t$, it has access only to the tokens up to and including $t$. This is appropriate for language modeling, where you should not peek at future tokens, but it is suboptimal for many other tasks.

Consider named entity recognition: to correctly label "Washington" in a sentence, it helps enormously to see what comes after it. "Washington led the troops" vs. "Washington state has excellent wines" vs. "George Washington founded the nation" each suggest very different entity types. A left-to-right RNN can only use the left context; a bidirectional RNN captures both.

The bidirectional hidden state at position $t$ is:

$$\overrightarrow{\mathbf{h}}_t = \tanh\!\left(\mathbf{W}_{hh}^{(f)}\,\overrightarrow{\mathbf{h}}_{t-1} + \mathbf{W}_{xh}^{(f)}\,\mathbf{x}_t + \mathbf{b}_h^{(f)}\right)$$ $$\overleftarrow{\mathbf{h}}_t = \tanh\!\left(\mathbf{W}_{hh}^{(b)}\,\overleftarrow{\mathbf{h}}_{t+1} + \mathbf{W}_{xh}^{(b)}\,\mathbf{x}_t + \mathbf{b}_h^{(b)}\right)$$ $$\mathbf{h}_t^{\text{bi}} = \left[\overrightarrow{\mathbf{h}}_t;\,\overleftarrow{\mathbf{h}}_t\right]$$

The combined representation $\mathbf{h}_t^{\text{bi}} \in \mathbb{R}^{2H}$ encodes context from both directions. The forward and backward RNNs have independent parameters, so the total parameter count roughly doubles compared to a unidirectional RNN with the same hidden size.

Bidirectional RNNs are widely used for:

• Named entity recognition and POS tagging: token-level classification tasks that benefit from full sequence context
• Question answering: encoding a question or context passage where both left and right context matter for each token
• Reading comprehension: attending over a passage where understanding each span requires knowledge of the whole document
• Pre-training language representations: BERT and ELMo use bidirectional architectures (though BERT uses transformers rather than RNNs)

The key limitation of bidirectional RNNs is that they require the entire sequence to be available before processing begins. They cannot be used for streaming or online scenarios where you must process and respond to tokens as they arrive. For generation tasks, unidirectional RNNs are necessary.

In PyTorch, nn.RNN supports bidirectionality through the bidirectional=True parameter. The output at each time step has shape (batch, seq_len, 2 * hidden_size) when bidirectional, where the two halves correspond to the forward and backward hidden states.