Bidirectional RNNs › Limitations of Bidirectional RNNs

Bidirectional RNNs are powerful for encoding tasks, but they have limitations.

The most fundamental limitation is that bidirectionality is incompatible with left-to-right text generation. When generating the next token in a sequence, future tokens do not exist yet. There is no right context to process in the backward direction. Attempting to generate text autoregressively with a BiRNN requires producing the entire sequence before the backward pass can run, which makes it impossible to condition generation on previous outputs.

This is why language models like GPT use strictly unidirectional (causal) architectures: the model is only permitted to attend to prior context. Bidirectionality is reserved for encoders. In the encoder-decoder sequence-to-sequence framework (covered in Part XII), a BiLSTM or BiGRU is commonly used as the encoder (which reads the full input sequence), while the decoder uses a unidirectional LSTM or GRU (which generates output tokens one at a time).

This architectural split between bidirectional encoders and unidirectional decoders is a consequence of a deep constraint. Every conditional generation system, whether seq2seq, BERT, or an image captioning model, must eventually commit to producing tokens in a fixed order. The encoder can be as bidirectional and contextually rich as desired, but the decoder must be causal.

Bidirectionality doubles the output dimension from $d$ to $2d$. When you feed BiRNN outputs into a downstream layer, that layer's input dimension is twice as large. For a classification head, this means a weight matrix that is twice as wide. For stacked BiRNNs, this doubles the input size of each successive layer relative to a unidirectional stack with the same hidden dimension. Memory and compute scale accordingly.

In practice, practitioners often halve the hidden dimension when switching to bidirectional models to keep the total parameter count comparable: a BiLSTM with $d=256$ has similar capacity to a unidirectional LSTM with $d=512$ but with different information routing. However, this equivalence is approximate: the bidirectional model distributes its capacity across two independent passes, while the unidirectional model concentrates it in one. For different tasks and datasets, either distribution can be preferable.

A unidirectional RNN can process a sequence incrementally: you receive token 1, compute $h_1$, receive token 2, compute $h_2$, and so on. You can produce outputs immediately after each token arrives. This is valuable in real-time applications, such as transcribing speech as it is spoken or processing log events as they stream in.

A bidirectional RNN must have the entire sequence before the backward pass can begin. There is no way to compute $\overleftarrow{h}_1$ until you have seen the last token. This introduces a latency of the full sequence length, making bidirectionality impractical for true streaming or online inference.

Workarounds exist but are costly. Chunk-based bidirectionality processes the sequence in overlapping windows, applying bidirectionality within each window but not across windows. This reduces latency at the cost of inter-chunk context. Another approach uses look-ahead buffers: wait for a fixed number of future tokens before processing the current token. This reduces but does not eliminate latency. None of these approximations match the quality of true full-sequence bidirectionality.

In temporal sequence prediction, including forecasting stock prices from past prices, predicting the next event in a clinical timeline, or modeling traffic patterns, conditioning on future information would constitute data leakage. Bidirectionality must be avoided or carefully scoped in such settings, since the backward pass sees information that would not be available at prediction time.

This is a subtle trap for practitioners. A model trained with bidirectionality on a temporal dataset will appear to perform excellently during training and validation if those splits do not carefully respect temporal ordering. But at deployment, the future tokens are unavailable, and the model's bidirectional representations are meaningless. The lesson is that architecture choices must reflect the inference-time information availability of the task.

Both the forward and backward passes of a BiRNN are sequential: each step depends on the previous step, so the computation for a sequence of length $T$ requires $T$ serial steps in each direction. This is the fundamental scalability limitation of all RNN architectures, bidirectional or not. Transformers overcome this with parallel self-attention that processes all positions simultaneously, which is one of the main reasons transformers scaled more efficiently to large models and large datasets than BiLSTMs.

For short sequences (up to a few hundred tokens), the sequential computation of BiLSTMs is rarely a bottleneck in practice: modern hardware can process them quickly. For long sequences (thousands of tokens), the sequential overhead becomes significant, and transformers have a clear advantage.

From a historical perspective, bidirectional RNNs represented a major advance in NLP. BiLSTM-CRF models dominated NER benchmarks for several years, and BiLSTMs formed the encoder component of state-of-the-art machine translation systems. But they have largely been superseded by transformer-based encoders, particularly BERT.

BERT uses self-attention rather than recurrence, which allows it to be bidirectional by design: every token attends to every other token directly in a single forward pass. The per-token representations produced by BERT are conceptually similar to BiLSTM outputs: each token embedding encodes full left and right context. But the attention mechanism achieves this more efficiently (no sequential computation) and with greater modeling capacity (every pair of tokens can interact directly, regardless of distance, without information bottlenecks in a hidden state). We will explore BERT and self-attention in detail later in the book.

This does not diminish the importance of understanding bidirectional RNNs. The conceptual insight, that encoding tasks benefit from full-context representations, carries directly from BiLSTMs to transformers. BERT's masked language modeling objective is specifically designed to force the model to use bidirectional context. The transition from BiLSTMs to BERT was an architectural evolution, not a conceptual revolution.