Bidirectional RNNs › What Each Direction Contributes

It is worth thinking carefully about what information each directional pass captures.

The forward hidden state $\overrightarrow{h}_t$ is an encoding of the prefix up to position $t$. For a sentence like "The bank by the river was flooded," after processing "The bank by the river", the forward state has absorbed the ambiguity of "bank" and the clarifying phrase "by the river." By the time the forward pass reaches "was," it knows this is probably a geographical bank.

The backward hidden state $\overleftarrow{h}_t$ is an encoding of the suffix starting at position $t$. Processing the same sentence from right to left, the backward state for "bank" has already seen "was flooded" and has strong evidence about the meaning. The backward pass essentially previews the consequences of each word before committing to a representation.

Concatenating them at "bank" gives a representation that knows both the context leading up to "bank" and the consequences following it. This full-context representation is exactly what a sequence labeler needs when assigning a POS tag or NER label.

The forward and backward passes are not symmetric in practice, even though they are structurally identical. Natural language has strong left-to-right dependencies: subjects tend to precede verbs, which tend to precede objects, and this ordering creates predictive structure that the forward LSTM can exploit. The backward pass processes language in its "unnatural" order, so it sees less predictive structure from that direction. The backward LSTM therefore tends to act more as a global context signal, less as a precise local predictor.

Research into what each direction learns has found that the forward direction often captures syntactic dependencies within phrases (determining whether the current token is a head or a modifier of the previous phrase), while the backward direction often captures higher-level semantic context (what broad topic is being discussed, what the overall sentence is about). When these representations are concatenated, the model has access to both fine-grained local syntax and broader semantic framing simultaneously.

This asymmetry is also visible in ablation studies. For most NLP tasks, removing the backward pass hurts performance more than removing the forward pass. Early tokens in sentences carry heavy disambiguation burdens, and those early tokens primarily benefit from backward context (since they have little forward context). The loss of the backward pass is therefore felt most acutely at the beginning of sequences, which is often where the most ambiguous tokens appear.