Bidirectional RNNs › The Problem with Unidirectional Processing

Before examining how bidirectional RNNs work, it helps to understand exactly when unidirectionality hurts and why the solution is not simply training the model longer or making it deeper.

When a unidirectional RNN processes a sequence, the hidden state at each position $t$ is a function of all preceding tokens:

$$h_t = f(h_{t-1}, x_t)$$

where:

• $h_t$ is the hidden state at position $t$
• $h_{t-1}$ is the previous hidden state
• $x_t$ is the input at position $t$
• $f$ is the recurrent function (tanh cell, LSTM cell, GRU cell, etc.)

This means that when the model assigns a label to position $t$, it makes its decision based only on $x_1, x_2, \ldots, x_t$. Future tokens $x_{t+1}, x_{t+2}, \ldots, x_T$ are invisible.

In many NLP tasks, this is a severe handicap. Consider these examples:

• Named entity recognition: Determining whether "Apple" in a sentence refers to the fruit or the company often requires seeing the verb or object that follows.
• Part-of-speech tagging: "I can fish" versus "I can fish tomorrow." The word "can" is either a modal verb or a noun, and the context on both sides determines the answer.
• Semantic role labeling: Identifying who did what to whom requires seeing the full clause structure.
• Coreference resolution: Deciding that "he" refers to "John" sometimes requires reading several words ahead.

The key insight is that these are all encoding tasks: the input sequence is fully available, and the goal is to produce a rich representation of it. This is fundamentally different from generation tasks like language modeling, where the future is unknown.

The distinction between encoding and generation tasks runs throughout natural language processing, and it matters deeply for architecture selection.

In a generation task, the model produces outputs one token at a time, conditioned on all previously generated tokens. Language modeling, machine translation output, and dialogue response generation all fit this pattern. Future outputs do not exist yet when the model is making each decision, so left-to-right processing is required because future outputs do not yet exist. Any architecture that peeks at future tokens during generation would be cheating by conditioning on information that does not exist.

In an encoding task, the model reads an entire input sequence and produces a representation for each position (or a single representation for the whole sequence). Named entity recognition, part-of-speech tagging, dependency parsing, reading comprehension, and semantic textual similarity are all encoding tasks. Here, all tokens are simultaneously available before any output is produced. Restricting such a model to unidirectional processing discards information that is right there in the input and could improve every prediction.

This distinction also explains why the dominant pre-transformer architectures often split into two families: left-to-right language models (ELMo, GPT) for generation-heavy tasks, and bidirectional encoders (BiLSTM-based models, eventually BERT) for understanding-heavy tasks. The transformer era unified these under a single framework, but the underlying tension between generation and encoding remains.

A question that sometimes arises is: if a unidirectional LSTM can capture long-range dependencies through its cell state, why does it need a backward pass? Why not just train longer and let it learn to anticipate future context implicitly?

The answer is that anticipation requires the model to predict future inputs from past inputs, which is a fundamentally harder problem than observing them directly. A sufficiently powerful unidirectional LSTM can, in principle, learn statistical associations between current tokens and future tokens when those associations are strong and consistent. For a few reliable patterns, this works reasonably well. But the real world is full of local, idiosyncratic, and unpredictable context. "Washington" followed by "DC" versus "Washington" followed by "Nationals" versus "Washington" followed by "Irving" are all different entity types, and no amount of left-context processing can reliably predict which follows.

The backward pass is not a soft, probabilistic anticipation. It is an exact, deterministic observation of the future tokens, encoded into the representation. The backward hidden state at position $t$ has processed $x_{t+1}, x_{t+2}, \ldots, x_T$. No amount of longer training on a unidirectional model gives it direct access to those tokens at prediction time.

There is also an information-theoretic argument. A unidirectional model at position $t$ receives exactly $t$ tokens of information: $x_1$ through $x_t$. A bidirectional model at position $t$ receives $T$ tokens of information: all tokens from both directions. For large $T$ and small $t$ (early positions in a long sequence), the information deficit of the unidirectional model is especially severe. The first token in a 100-word sentence has almost no forward context at all. The backward pass repairs this by ensuring the first token's representation has been informed by all 99 tokens that follow it.