Bidirectional RNNs: Full-Sequence Context for NLP

Every RNN architecture we have covered so far, from the vanilla RNN to the GRU, processes sequences in one direction: left to right. The network sees the first token, updates its hidden state, moves to the second token, and so on. By the time the model assigns a label to the fifth token in a sentence, it has seen tokens one through five, but nothing from tokens six onward.

This unidirectional constraint is a natural fit for language generation. When you are predicting the next word, you cannot look at future tokens because they have not been produced yet. But it creates a real problem for tasks where you already have the full sequence and your job is to analyze or label it. Consider named entity recognition: if you are deciding whether "Washington" in a sentence refers to a person or a city, the words that come after "Washington" often contain the decisive clue. "Washington visited the capital" and "Washington signed the treaty" present very different contexts, and the word "visited" or "signed" can only resolve the ambiguity if the model is allowed to look right as well as left.

Bidirectional RNNs solve this by running two separate passes over the sequence: one in the forward direction (left to right) and one in the backward direction (right to left). At each position, the representations from both passes are concatenated, giving every token a view of its entire surrounding context. This seemingly simple modification, introduced by Schuster and Paliwal in 1997, substantially improved sequence labeling performance and became the foundation for most pre-transformer NLP architectures.

The core insight is about the type of task you are solving. Some tasks require you to produce output before seeing all the input, and some tasks let you see the entire input before producing any output. For the second class of tasks, there is no reason to artificially limit the model to processing in one direction. Bidirectionality takes full advantage of the complete input, and the empirical gains are substantial.

Before examining how bidirectional RNNs work, it helps to understand exactly when unidirectionality hurts and why the solution is not simply training the model longer or making it deeper.

When a unidirectional RNN processes a sequence, the hidden state at each position $t$ is a function of all preceding tokens:

where:

• $h_t$ is the hidden state at position $t$
• $h_{t-1}$ is the previous hidden state
• $x_t$ is the input at position $t$
• $f$ is the recurrent function (tanh cell, LSTM cell, GRU cell, etc.)

This means that when the model assigns a label to position $t$, it makes its decision based only on $x_1, x_2, \ldots, x_t$. Future tokens $x_{t+1}, x_{t+2}, \ldots, x_T$ are invisible.

In many NLP tasks, this is a severe handicap. Consider these examples:

• Named entity recognition: Determining whether "Apple" in a sentence refers to the fruit or the company often requires seeing the verb or object that follows.
• Part-of-speech tagging: "I can fish" versus "I can fish tomorrow." The word "can" is either a modal verb or a noun, and the context on both sides determines the answer.
• Semantic role labeling: Identifying who did what to whom requires seeing the full clause structure.
• Coreference resolution: Deciding that "he" refers to "John" sometimes requires reading several words ahead.

The key insight is that these are all encoding tasks: the input sequence is fully available, and the goal is to produce a rich representation of it. This is fundamentally different from generation tasks like language modeling, where the future is unknown.

The distinction between encoding and generation tasks runs throughout natural language processing, and it matters deeply for architecture selection.

In a generation task, the model produces outputs one token at a time, conditioned on all previously generated tokens. Language modeling, machine translation output, and dialogue response generation all fit this pattern. Future outputs do not exist yet when the model is making each decision, so left-to-right processing is required because future outputs do not yet exist. Any architecture that peeks at future tokens during generation would be cheating by conditioning on information that does not exist.

In an encoding task, the model reads an entire input sequence and produces a representation for each position (or a single representation for the whole sequence). Named entity recognition, part-of-speech tagging, dependency parsing, reading comprehension, and semantic textual similarity are all encoding tasks. Here, all tokens are simultaneously available before any output is produced. Restricting such a model to unidirectional processing discards information that is right there in the input and could improve every prediction.

This distinction also explains why the dominant pre-transformer architectures often split into two families: left-to-right language models (ELMo, GPT) for generation-heavy tasks, and bidirectional encoders (BiLSTM-based models, eventually BERT) for understanding-heavy tasks. The transformer era unified these under a single framework, but the underlying tension between generation and encoding remains.

A question that sometimes arises is: if a unidirectional LSTM can capture long-range dependencies through its cell state, why does it need a backward pass? Why not just train longer and let it learn to anticipate future context implicitly?

The answer is that anticipation requires the model to predict future inputs from past inputs, which is a fundamentally harder problem than observing them directly. A sufficiently powerful unidirectional LSTM can, in principle, learn statistical associations between current tokens and future tokens when those associations are strong and consistent. For a few reliable patterns, this works reasonably well. But the real world is full of local, idiosyncratic, and unpredictable context. "Washington" followed by "DC" versus "Washington" followed by "Nationals" versus "Washington" followed by "Irving" are all different entity types, and no amount of left-context processing can reliably predict which follows.

The backward pass is not a soft, probabilistic anticipation. It is an exact, deterministic observation of the future tokens, encoded into the representation. The backward hidden state at position $t$ has processed $x_{t+1}, x_{t+2}, \ldots, x_T$. No amount of longer training on a unidirectional model gives it direct access to those tokens at prediction time.

There is also an information-theoretic argument. A unidirectional model at position $t$ receives exactly $t$ tokens of information: $x_1$ through $x_t$. A bidirectional model at position $t$ receives $T$ tokens of information: all tokens from both directions. For large $T$ and small $t$ (early positions in a long sequence), the information deficit of the unidirectional model is especially severe. The first token in a 100-word sentence has almost no forward context at all. The backward pass repairs this by ensuring the first token's representation has been informed by all 99 tokens that follow it.

A bidirectional RNN runs two independent recurrent networks over the same input sequence.

The forward network processes tokens in the standard left-to-right order:

The backward network processes the same tokens in reverse order, from position $T$ down to position $1$:

where:

• $\overrightarrow{h}_t$ is the forward hidden state at position $t$, summarizing context from the left (tokens $1$ through $t$)
• $\overleftarrow{h}_t$ is the backward hidden state at position $t$, summarizing context from the right (tokens $T$ down through $t$)
• $f$ is the same type of recurrent cell used in both directions, though the two networks have separate, independent parameters

At each position $t$, the two hidden states are concatenated to form a combined representation:

where:

• $[\cdot ; \cdot]$ denotes vector concatenation
• If each directional hidden state has dimension $d$, the combined state has dimension $2d$

This combined representation at position $t$ encodes information from the entire sequence. The forward component carries context from tokens to the left of $t$, and the backward component carries context from tokens to the right. Every position benefits from full sentence-level context.

A subtle but important point is that the two directional networks have completely independent parameters. The forward LSTM has its own input-to-hidden weights, hidden-to-hidden weights, and biases. The backward LSTM has a separate, unshared set of weights. They are trained jointly via backpropagation, but gradient updates to one set do not directly affect the other.

This independence is intentional and beneficial. The forward network is optimized to summarize left context effectively. The backward network is optimized to summarize right context effectively. These tasks differ: left context tends to encode the topic and the syntactic head of phrases that came before, while right context tends to encode the consequences and completions of the current phrase. Separate parameters let each direction specialize in what it does best. Sharing parameters would force both directions to use the same representational strategy, which would likely hurt both.

Both directional networks are initialized independently, typically with the same random initialization scheme applied separately. During training, gradients flow backward through each direction independently: gradients for the forward LSTM flow from right to left through time (from position $T$ back to position $1$), and gradients for the backward LSTM flow from left to right through time (from position $1$ up to position $T$). Both sets of gradients are accumulated and applied to their respective parameters at each update step.

The combined representation at each position receives gradient contributions from both the output layer (through the classification head or whatever follows) and is passed back to both directional networks. This means the forward LSTM's weights are shaped by how well the combined forward-plus-backward representation classifies each position, not just by how well the forward representation alone would classify it. The two networks are coupled through the shared output loss, even though their parameters are independent.

It is worth thinking carefully about what information each directional pass captures.

The forward hidden state $\overrightarrow{h}_t$ is an encoding of the prefix up to position $t$. For a sentence like "The bank by the river was flooded," after processing "The bank by the river", the forward state has absorbed the ambiguity of "bank" and the clarifying phrase "by the river." By the time the forward pass reaches "was," it knows this is probably a geographical bank.

The backward hidden state $\overleftarrow{h}_t$ is an encoding of the suffix starting at position $t$. Processing the same sentence from right to left, the backward state for "bank" has already seen "was flooded" and has strong evidence about the meaning. The backward pass essentially previews the consequences of each word before committing to a representation.

Concatenating them at "bank" gives a representation that knows both the context leading up to "bank" and the consequences following it. This full-context representation is exactly what a sequence labeler needs when assigning a POS tag or NER label.

The forward and backward passes are not symmetric in practice, even though they are structurally identical. Natural language has strong left-to-right dependencies: subjects tend to precede verbs, which tend to precede objects, and this ordering creates predictive structure that the forward LSTM can exploit. The backward pass processes language in its "unnatural" order, so it sees less predictive structure from that direction. The backward LSTM therefore tends to act more as a global context signal, less as a precise local predictor.

Research into what each direction learns has found that the forward direction often captures syntactic dependencies within phrases (determining whether the current token is a head or a modifier of the previous phrase), while the backward direction often captures higher-level semantic context (what broad topic is being discussed, what the overall sentence is about). When these representations are concatenated, the model has access to both fine-grained local syntax and broader semantic framing simultaneously.

This asymmetry is also visible in ablation studies. For most NLP tasks, removing the backward pass hurts performance more than removing the forward pass. Early tokens in sentences carry heavy disambiguation burdens, and those early tokens primarily benefit from backward context (since they have little forward context). The loss of the backward pass is therefore felt most acutely at the beginning of sequences, which is often where the most ambiguous tokens appear.

Nothing prevents you from using LSTM or GRU cells as the recurrent unit in each direction. In fact, bidirectional LSTMs (BiLSTMs) and bidirectional GRUs (BiGRUs) are far more common in practice than bidirectional vanilla RNNs, because they combine the long-range dependency handling of LSTM/GRU gates with the full-context encoding of bidirectionality.

A BiLSTM runs two independent LSTM networks over the sequence. Each direction maintains its own cell state $c_t$ and hidden state $h_t$:

Forward LSTM:

Backward LSTM:

The combined output is:

where:

• $\overrightarrow{h}_t$ and $\overleftarrow{h}_t$ are the forward and backward LSTM hidden states at position $t$
• The cell states $\overrightarrow{c}_t$ and $\overleftarrow{c}_t$ are internal to each LSTM direction and are not concatenated for output

A BiGRU follows the same pattern with GRU cells instead of LSTM cells. Because GRUs have no separate cell state, only hidden states are concatenated. BiGRUs are slightly simpler and often competitive with BiLSTMs on many benchmarks while requiring fewer parameters.

The choice between BiLSTM and BiGRU typically comes down to the same considerations as the unidirectional case: dataset size, sequence length, and whether the full gating machinery of an LSTM is warranted given your compute budget.

Using LSTM or GRU cells in each direction amplifies the benefits of bidirectionality. A vanilla bidirectional RNN still suffers from vanishing gradients within each directional pass: the forward hidden state at position $T$ may contain very little information about position 1 if the sequence is long. This means that even though the representation at position 1 incorporates backward context, the backward context itself may be poorly encoded if the backward pass has trouble maintaining information over many steps.

LSTM cells address this precisely. Each directional pass in a BiLSTM can maintain information over long distances through its cell state, so the backward hidden state at position 1 contains well-preserved information from position $T$, even in sequences of hundreds of tokens. The combination of LSTM's temporal memory and bidirectionality's directional completeness is greater than either alone: you get good long-range memory in both directions simultaneously.

This is why BiLSTMs, rather than bidirectional vanilla RNNs, became the dominant architecture for NLP before the transformer era. Vanilla bidirectional RNNs are mostly of theoretical interest now.

Bidirectional processing was designed for and excels at sequence labeling, where the model assigns a label to every token in the input.

In NER, the model reads a sentence and assigns each word a label such as B-PER (beginning of a person entity), I-PER (inside a person entity), B-ORG, I-ORG, O (outside any entity), and so on, following the BIO scheme covered in Part VI.

A BiLSTM is particularly well-suited for NER because entity boundaries often depend on both preceding and following context. The surname "Jordan" could be a person, a country, or a river. The context before it might disambiguate nothing (subject position is neutral), but the context after it, such as "scored 38 points" versus "borders Iraq", makes the entity type unambiguous.

A standard BiLSTM-CRF model for NER passes the BiLSTM output through a CRF layer that enforces globally consistent label sequences. The BiLSTM handles the contextual representation; the CRF handles the structured output constraint (you cannot have an I-PER tag immediately after an O tag, for example). This combination, introduced by Lample et al. in 2016, became the dominant NER architecture before BERT.

The division of labor here is clean and important. The BiLSTM produces, at each position, a vector of scores over all possible tags: how likely is each of B-PER, I-PER, B-ORG, I-ORG, O, and so on? These scores are locally informed but globally unconstrained. The CRF layer then finds the globally optimal tag sequence over all positions jointly, subject to learned transition costs between tags. It can encode knowledge like "I-LOC can only follow B-LOC or I-LOC, never O or B-PER." This global inference step catches inconsistencies that would slip through a per-position softmax. For instance, the per-position softmax might confidently predict I-ORG for position 3 even if position 2 was tagged O, because it only sees position 3 in isolation. The CRF prevents this by considering the whole sequence at once.

The BiLSTM-CRF paper reported state-of-the-art results on the CoNLL 2003 NER benchmark with a model that also incorporated character-level representations. Character-level BiLSTMs were used to encode each word's spelling. This provides morphological information (prefixes, suffixes, capitalization) that word-level embeddings miss. The spelling of "McCartney" provides strong evidence for a person entity even without context, but a word-level model would not know this for an out-of-vocabulary name it has never seen. The combination of word embeddings, character-level BiLSTMs, sentence-level BiLSTMs, and a CRF output layer formed a complete system that dominated NER research for several years.

POS tagging assigns grammatical categories (noun, verb, adjective, preposition, determiner, etc.) to each word. Like NER, it benefits from seeing both sides of a word. The sentence "They can fish in the lake" requires knowing that "can" is followed by "fish" (another potential noun or verb) and that "fish" is followed by "in the lake" (a prepositional phrase indicating location, not the action of swimming). The backward pass on a BiRNN provides exactly this right-side context before any forward-direction decisions are made.

POS tagging also illustrates an important property of bidirectionality: it improves the accuracy of ambiguous tokens and the confidence of unambiguous ones. Even when the forward context is sufficient to determine the correct tag, having the backward context as corroborating evidence produces a more confident and stable representation. This reduces variance across different sentences and leads to better generalization.

Noun phrase chunking, covered in Part VI, also benefits from bidirectionality. Determining where a chunk begins and ends requires both knowing what came before (whether we are inside a noun phrase) and what comes next (whether the next token opens a new phrase or continues the current one).

Chunking is essentially a boundary detection problem layered on top of sequence labeling. Each token receives a label indicating its role within a chunk: B (beginning of chunk), I (inside chunk), or O (outside all chunks). The precise definition of chunk boundaries often requires seeing several tokens in both directions simultaneously. For example, identifying the end of a noun phrase in "the very large blue car near the corner" is much easier with backward context: "near" signals the end of the first noun phrase and the beginning of a new prepositional phrase.

More broadly, any task where the correct output for a token depends on the surrounding sequence rather than just the prefix benefits from bidirectionality. This covers a wide swath of classic NLP pipelines: semantic role labeling, coreference mention detection, relation extraction, and even reading comprehension tasks where spans must be identified within a passage. The common thread is that the full context is available at inference time, so there is no reason to artificially restrict the model to left context only.

Semantic role labeling (SRL) assigns argument roles to spans in a sentence: who is the agent (the entity doing something), what is the patient (the entity being acted upon), what is the instrument, and so on. For the sentence "The chef in the kitchen prepared the meal with fresh ingredients," SRL must identify "The chef in the kitchen" as the agent of "prepared," "the meal" as the patient, and "fresh ingredients" as the instrument.

SRL is especially challenging for unidirectional models because the predicate (the main verb) often appears in the middle of the sentence, and arguments can appear both before and after it. A model that has only seen tokens up to the predicate has no information about what arguments follow it. A BiLSTM has full information about both preceding and following arguments when computing the representation of the predicate, which makes argument identification significantly more accurate.

Coreference resolution identifies when different expressions in a text refer to the same entity. Determining that "she" in the second sentence of a paragraph refers to "Dr. Nguyen" introduced in the first sentence requires reading forward through the text. But within a single sentence, the antecedent for a pronoun often appears later: "After she finished the experiment, Dr. Nguyen wrote up the results." Here "she" refers to "Dr. Nguyen," which appears after the pronoun. A unidirectional model reading left-to-right would have no information about "Dr. Nguyen" when it processes "she." A BiLSTM's backward pass has already processed "Dr. Nguyen" before encoding "she," making the coreference link much easier to identify.

Let's trace through a simple sentence to make the mechanics concrete.

Consider the four-word sentence: "Banks near the river"

We want to assign a POS tag to each word. The answer is NOUN, ADP (preposition), DET, NOUN.

Forward pass (left to right):

The forward LSTM starts with $\overrightarrow{h}_0 = \mathbf{0}$ and processes:

• Position 1, "Banks": $\overrightarrow{h}_1$ encodes "Banks" with no prior context. The representation is ambiguous because "banks" can be a noun (financial institutions, riverbanks) or a verb (to bank, to store).
• Position 2, "near": $\overrightarrow{h}_2$ encodes "near" given "Banks." The forward context "Banks near" strengthens the noun reading for "Banks" and establishes "near" as a preposition.
• Position 3, "the": $\overrightarrow{h}_3$ encodes "the" given "Banks near." The definite article after a preposition signals an upcoming noun phrase.
• Position 4, "river": $\overrightarrow{h}_4$ encodes "river" given the full left context. The right context is now complete.

Backward pass (right to left):

The backward LSTM starts with $\overleftarrow{h}_5 = \mathbf{0}$ (past the last token) and processes:

• Position 4, "river": $\overleftarrow{h}_4$ encodes "river" with no right context.
• Position 3, "the": $\overleftarrow{h}_3$ encodes "the river" from the right. The article-noun combination encodes cleanly.
• Position 2, "near": $\overleftarrow{h}_2$ encodes "near the river" from the right. The preposition-noun phrase combination establishes a locative prepositional phrase.
• Position 1, "Banks": $\overleftarrow{h}_1$ encodes "Banks" given "near the river" from the right, which is strong evidence for the noun reading. The locative context "near the river" does not follow verbs, only nouns.

Combined representations:

At each position, the forward and backward states are concatenated:

• "Banks": $[\overrightarrow{h}_1; \overleftarrow{h}_1]$ carries right context "near the river," providing strong evidence for NOUN
• "near": $[\overrightarrow{h}_2; \overleftarrow{h}_2]$ carries left context "Banks" and right context "the river," confirming ADP
• "the": $[\overrightarrow{h}_3; \overleftarrow{h}_3]$ carries both directions, confirming DET
• "river": $[\overrightarrow{h}_4; \overleftarrow{h}_4]$ carries left context "Banks near the," confirming NOUN

Each combined representation carries enough information to assign the correct POS tag with high confidence, even for the ambiguous "Banks" at position 1.

Notice that the ambiguity of "Banks" is resolved entirely by the backward context. If you removed the backward pass and gave the forward pass only, position 1 would have to make a decision based solely on the word "Banks" itself, with no surrounding evidence. The backward pass is what makes this case tractable.

Consider a longer and more realistic example where multiple tokens are ambiguous simultaneously: "The lead pipe in the kitchen was painted lead-colored."

This sentence contains two instances of "lead": the first as a noun (lead pipe, the heavy metal), and the second as an adjective (lead-colored). A POS tagger must assign NOUN to the first and ADJ to the second.

For the first "lead," the forward context at that position is just "The," which tells us nothing about whether "lead" is a noun or verb ("They lead the parade") or adjective. The backward context includes "pipe in the kitchen was painted lead-colored," which establishes a long noun phrase and makes the noun reading unambiguous.

For the second "lead," the forward context "...was painted" indicates an adjective or past participle is expected, and the backward context "colored" (in the compound adjective "lead-colored") confirms the adjectival reading.

Neither of these disambiguations is easy for a unidirectional model. Both are straightforward for a BiLSTM with full bidirectional context.

PyTorch makes bidirectional RNNs straightforward with a single constructor argument. Let's build a BiLSTM sequence tagger and apply it to a POS tagging task.

We start by importing dependencies and preparing a small toy dataset for demonstration.

We have a small vocabulary built from 6 training sentences. In a real system you would use hundreds of thousands of sentences, but this is sufficient to demonstrate the architecture.

The bidirectional=True parameter in nn.LSTM is all that changes from a standard LSTM. PyTorch internally creates two LSTM layers and concatenates their outputs.

Notice the output shape: (batch=1, seq_len=4, num_tags=5). The model produces a tag distribution for every position in the sequence simultaneously, which is what sequence labeling means: one prediction per input token, not one prediction per sequence.

The output tensor from a bidirectional LSTM in PyTorch deserves careful examination. When bidirectional=True, the lstm_out tensor at each position is the concatenation of the forward hidden state and the backward hidden state. Specifically, lstm_out[b, t, :hidden_dim] is the forward hidden state at position t in batch element b, and lstm_out[b, t, hidden_dim:] is the backward hidden state at position t.

This means the first half of the output vector at each position carries information about all tokens to the left (and the current token), and the second half carries information about all tokens to the right (and the current token). The classifier layer, a simple linear projection, takes this concatenated vector and maps it to tag scores. In a real system, you might use a more sophisticated output layer, but even a linear classifier applied independently to each position's concatenated representation is often highly effective because the BiLSTM has already done the hard work of context integration.

The final hidden states, which PyTorch returns as the second output of lstm(), work slightly differently under bidirectionality. PyTorch returns them with shape (num_directions * num_layers, batch, hidden_dim). For a single-layer bidirectional LSTM, this is shape (2, batch, hidden_dim), where index 0 contains the forward LSTM's final hidden state (at position $T$) and index 1 contains the backward LSTM's final hidden state (at position $1$). If you want a single fixed-length representation for the whole sequence, a common approach is to concatenate these two terminal states.

The key shape difference from a unidirectional LSTM is that the output dimension is hidden_dim * 2.

A bidirectional LSTM with hidden dimension $d$ is not twice as expensive as a unidirectional LSTM with the same $d$. If you want to match the unidirectional model's capacity, use a bidirectional model with hidden dimension $d/2$ for each direction, achieving comparable parameter counts while still gaining full-context representations.

The key parameters for nn.LSTM and nn.GRU when using bidirectionality are:

• bidirectional: Set to True to run both directions. When True, the module internally creates two recurrent networks with independent weight matrices.
• hidden_size: The dimension of each directional hidden state. The output tensor's last dimension will be hidden_size * 2 because the two directions are concatenated.
• num_layers: Number of stacked BiRNN layers. Each layer takes the concatenated output of the previous layer as input to both its forward and backward networks.
• batch_first: Set to True so tensors have shape (batch, seq_len, features) rather than (seq_len, batch, features).

A natural extension of the single-layer BiRNN is to stack multiple bidirectional layers vertically, creating a deep bidirectional architecture. In a stacked BiRNN with $L$ layers, the output of each layer (the concatenated forward and backward hidden states at each position) becomes the input to the next layer above it. This creates a hierarchy of representations: lower layers tend to capture fine-grained local patterns, while upper layers capture broader syntactic and semantic structure.

The mechanics of stacking are straightforward in PyTorch: setting num_layers=2 in an nn.LSTM with bidirectional=True creates two stacked BiLSTM layers. Each layer operates on the full sequence with its own independent pair of forward and backward LSTMs. The output of the first layer, with shape (batch, seq_len, 2 * hidden_dim), becomes the input to the second layer, where the 2 * hidden_dim is the input dimension. The second layer also produces outputs with shape (batch, seq_len, 2 * hidden_dim).

This means the input dimension to the second layer is twice the hidden dimension of the first layer. If the first layer has hidden size 128, its output is 256-dimensional (128 from the forward pass, 128 from the backward pass), and the second layer's LSTM cells take 256-dimensional inputs. This is correct behavior, but it is easy to accidentally create parameter mismatches if you are constructing stacked layers manually.

Each layer in a stacked BiRNN can develop qualitatively different representations. In studies of deep BiLSTMs trained for language understanding tasks, the pattern is fairly consistent: lower layers specialize in morphological and syntactic information (word types, phrase boundaries, grammatical relationships), while upper layers specialize in semantic and pragmatic information (entity types, argument roles, discourse coherence).

This hierarchy is not imposed by the architecture: it emerges from training. The reason it emerges is that semantic information often requires syntactic information as a prerequisite. To determine that "Apple" is a company rather than a fruit, you first need to know that "Apple" is a noun (syntactic), then that it is a proper noun, then that the surrounding context is corporate or technological (semantic). The lower layers of a stacked BiLSTM naturally handle the earlier steps in this reasoning chain, and the upper layers handle the later steps.

Deeper architectures are more expressive but also more prone to overfitting. Several regularization strategies are commonly applied to stacked BiRNNs:

Dropout between layers is the most common. After the output of each layer (but before feeding it into the next), a dropout mask is applied, randomly zeroing some dimensions. PyTorch's nn.LSTM supports this directly via the dropout parameter, which applies dropout to all layers except the last.

Variational dropout (also called recurrent dropout) applies the same dropout mask consistently across all time steps within a single sequence, rather than sampling a new mask at each step. Standard dropout applied at each time step disrupts the recurrent state's temporal coherence; variational dropout preserves it while still providing regularization. This approach was popularized by Gal and Ghahramani in 2016 and is commonly used in high-performance BiLSTM systems.

Layer normalization can be applied within each LSTM cell to stabilize training in very deep stacks. This is less common than dropout but useful when training very deep models or when batch statistics are unreliable (small batches or very variable sequence lengths).

This section presents three visualizations: the per-position context coverage of a bidirectional pass, the parameter count tradeoff between unidirectional and bidirectional models at different hidden sizes, and the training loss curve from the POS tagger above.

To make the information flow concrete, let's visualize how much each directional pass contributes to the representation at each position in a sample sentence.

The chart shows why position "Banks" (position 1) particularly benefits from the backward pass: the forward pass has seen only 1 token (itself), while the backward pass has seen all 4 tokens from right to left. After concatenation, the representation of "Banks" has access to the full 4-token sentence context.

Doubling the output dimension by going bidirectional does not have to mean doubling the parameter count. If you halve the per-direction hidden size, the total stays comparable. The following chart shows total parameter counts for unidirectional and bidirectional LSTMs across a range of hidden sizes, using our toy vocabulary and tag set.

The matched-capacity BiLSTM (dashed green) closely tracks the unidirectional model's parameter count while still providing full bidirectional context at each position. This is the typical trade-off practitioners use: to switch a unidirectional model to bidirectional without increasing total parameters, halve the hidden dimension.

Let's also look at how training progresses for our BiLSTM tagger.

So far we have used the BiRNN for sequence labeling, where each position in the output is used independently. But what if you want a single fixed-length vector representing the entire sentence? This is needed for sentence classification, textual entailment, or any task where the output is a single label for the whole input.

There are several common strategies for reducing a sequence of per-token representations to a single vector:

The most direct approach uses the final hidden states from each direction. For a BiLSTM processing a sequence of length $T$:

• The forward LSTM produces its most informative final state at position $T$, after seeing all tokens.
• The backward LSTM produces its most informative final state at position $1$, after having processed the sequence from right to left.

Concatenating these gives a vector of size $2d$ that contains the "summary" of the full sequence from both directions. This is the standard approach when using an LSTM encoder in sequence-to-sequence models: the encoder's final hidden states are concatenated and passed to the decoder as the initial context.

The limitation of this approach is that the final hidden states must compress the entire sequence into a vector of fixed size. For long sequences, important details from the middle of the sequence may be lost.

An alternative is to average all per-token hidden states:

where $h_t = [\overrightarrow{h}_t ; \overleftarrow{h}_t]$ is the combined bidirectional representation at position $t$.

Mean pooling gives every position equal weight in the final representation. This is simple and often effective for tasks where the relevant information is distributed across the sequence (topic classification, sentiment analysis over long reviews). Its weakness is that it does not give more weight to the most informative tokens, treating every position equally regardless of importance.

Max pooling takes the element-wise maximum across all positions:

for each dimension $j$. This selects the most strongly activated value in each dimension, effectively finding the tokens that most strongly express each feature. Max pooling is commonly used in sentence classification tasks because it automatically focuses on the most salient signals in the sequence rather than averaging them out.

The most expressive pooling strategy uses an attention mechanism to compute a weighted sum of per-token representations:

where the attention weights $\alpha_t$ are computed by a small network (often a single linear layer followed by softmax) that scores each token's relevance. This allows the model to learn which positions are most important for the task at hand, rather than treating all positions equally (mean pooling) or always selecting the maximum value (max pooling).

Attention-based pooling is conceptually related to the attention mechanisms in sequence-to-sequence models, which we cover in Part XII. The key difference is that pooling attention produces a fixed summary for the whole sequence, while sequence-to-sequence attention produces a different weighted summary for each output position.

The BiLSTM-CRF model introduced by Lample et al. (2016) deserves extended treatment because it became such a dominant architecture for NER and is a canonical example of how bidirectionality and structured prediction complement each other.

After the BiLSTM produces per-token tag score vectors, the simplest output strategy is to apply a softmax at each position independently and take the argmax. This approach, called greedy decoding, produces the locally most probable tag at each position but ignores dependencies between adjacent tags.

Tag dependencies are real and important. In BIO tagging, the tag sequence "O, I-PER" is invalid: you cannot have an inside-entity tag without a preceding beginning-entity tag. "B-PER, I-ORG" is also invalid: you cannot be inside a different entity type than the one you began. These constraints cannot be captured by position-wise softmax, which sees each position in isolation.

The conditional random field (CRF) layer addresses this by modeling the joint probability of the entire tag sequence, rather than the product of per-position probabilities. It adds a matrix of transition scores $A$ where $A_{ij}$ is the score of transitioning from tag $i$ at position $t$ to tag $j$ at position $t+1$. The score of a full tag sequence $y_1, y_2, \ldots, y_T$ is:

where:

• $P_{t, y_t}$ is the BiLSTM emission score for tag $y_t$ at position $t$
• $A_{y_t, y_{t+1}}$ is the transition score from tag $y_t$ to tag $y_{t+1}$

The model is trained to maximize the log-likelihood of the correct tag sequence. At inference time, the Viterbi algorithm finds the tag sequence with the highest score, efficiently searching over all possible sequences.

The transition matrix $A$ is initialized randomly and learned from data. For NER with BIO tags, the model quickly learns that transitions like "O to I-PER" should have very negative scores (essentially preventing that transition) while transitions like "B-PER to I-PER" should have high positive scores. Once learned, these constraints apply globally to every sentence.

The BiLSTM provides local evidence: what each position looks like in context. The CRF provides global structure: what tag sequences are coherent. The combination is more powerful than either component alone. The BiLSTM alone would make locally good but globally inconsistent decisions. The CRF alone, without the BiLSTM's rich contextual representations, would have only crude features to work with.

The Viterbi algorithm for CRF inference has time complexity $O(T \cdot K^2)$, where $T$ is the sequence length and $K$ is the number of tags. For typical NER tag sets (5 to 20 tags), this is very efficient: the quadratic dependence is on a small constant. For large tag sets (hundreds of tags, as in some semantic parsing applications), the CRF becomes computationally expensive and simpler approximations may be necessary.

Training is similarly efficient. The forward-backward algorithm computes the partition function needed for the CRF log-likelihood in $O(T \cdot K^2)$ time. For sequence lengths in the hundreds and tag sets in the tens, this adds negligible cost compared to the BiLSTM forward and backward passes.

The ELMo (Embeddings from Language Models) system, published by Peters et al. in 2018, brought bidirectional LSTMs to contextual word representation learning at scale. It is worth understanding ELMo's design in detail because it represents the culmination of the BiLSTM era and directly motivates the transformer-based models that followed.

ELMo uses a deep two-layer bidirectional LSTM trained on a large corpus using a combination of forward and backward language modeling objectives. The forward LM predicts each word given all preceding words; the backward LM predicts each word given all following words. These two objectives are trained jointly on the same corpus.

The key architectural choice is that ELMo exposes all intermediate LSTM layer representations, rather than only the top layer. For a two-layer BiLSTM, this means three vectors are available for each token: the character-CNN embedding (which encodes morphology), the first BiLSTM layer's output, and the second BiLSTM layer's output. When ELMo is used downstream, these three representations are combined using learned scalar weights:

where $s_k$ are softmax-normalized task-specific weights, $h_{t,k}$ is the $k$-th layer representation at position $t$, and $\gamma$ is a task-specific scalar. Different NLP tasks learn different weighting profiles.

The finding that different ELMo layers encode different linguistic properties was empirically validated through a series of probing experiments. When the scalar weights are examined after training on different tasks:

• Syntactic tasks (POS tagging, constituency parsing) assign high weight to lower BiLSTM layers, which capture local morphological and syntactic patterns.
• Semantic tasks (coreference, SRL, NER) assign higher weight to upper layers, which capture broader semantic relationships.

This layer-wise specialization emerged purely from training on language modeling: no explicit supervision about which layers should encode syntax versus semantics was provided. The hierarchy of linguistic abstraction emerged naturally from the architecture and training objective.

ELMo demonstrated that contextual representations dramatically outperform fixed word embeddings, establishing the paradigm of pretraining a large language model and fine-tuning it for downstream tasks. However, ELMo's BiLSTM architecture has inherent limitations that BERT's transformer architecture overcame.

The most fundamental limitation is that ELMo's bidirectionality is not truly joint. The forward and backward LSTMs are trained separately (each on its own language modeling objective) and then concatenated. The forward LSTM has never "seen" the backward representations during its own training; it is optimized to predict words from left context alone. Similarly, the backward LSTM is optimized to predict words from right context alone. The concatenation happens only at the representation level, not during learning.

BERT's self-attention mechanism, by contrast, is truly jointly bidirectional: every token attends to every other token simultaneously in a single operation. There is no separate forward and backward pass. The model is trained to predict masked tokens using context from both sides at once. This joint training produces qualitatively richer representations because the model learns to integrate left and right context in a unified way rather than concatenating separately trained unidirectional features.

This distinction explains why BERT substantially outperformed ELMo despite using broadly similar pretraining concepts. The joint bidirectionality of attention improved on the concatenated-LSTM approach.

Bidirectional RNNs are powerful for encoding tasks, but they have limitations.

The most fundamental limitation is that bidirectionality is incompatible with left-to-right text generation. When generating the next token in a sequence, future tokens do not exist yet. There is no right context to process in the backward direction. Attempting to generate text autoregressively with a BiRNN requires producing the entire sequence before the backward pass can run, which makes it impossible to condition generation on previous outputs.

This is why language models like GPT use strictly unidirectional (causal) architectures: the model is only permitted to attend to prior context. Bidirectionality is reserved for encoders. In the encoder-decoder sequence-to-sequence framework (covered in Part XII), a BiLSTM or BiGRU is commonly used as the encoder (which reads the full input sequence), while the decoder uses a unidirectional LSTM or GRU (which generates output tokens one at a time).

This architectural split between bidirectional encoders and unidirectional decoders is a consequence of a deep constraint. Every conditional generation system, whether seq2seq, BERT, or an image captioning model, must eventually commit to producing tokens in a fixed order. The encoder can be as bidirectional and contextually rich as desired, but the decoder must be causal.

Bidirectionality doubles the output dimension from $d$ to $2d$. When you feed BiRNN outputs into a downstream layer, that layer's input dimension is twice as large. For a classification head, this means a weight matrix that is twice as wide. For stacked BiRNNs, this doubles the input size of each successive layer relative to a unidirectional stack with the same hidden dimension. Memory and compute scale accordingly.

In practice, practitioners often halve the hidden dimension when switching to bidirectional models to keep the total parameter count comparable: a BiLSTM with $d=256$ has similar capacity to a unidirectional LSTM with $d=512$ but with different information routing. However, this equivalence is approximate: the bidirectional model distributes its capacity across two independent passes, while the unidirectional model concentrates it in one. For different tasks and datasets, either distribution can be preferable.

A unidirectional RNN can process a sequence incrementally: you receive token 1, compute $h_1$, receive token 2, compute $h_2$, and so on. You can produce outputs immediately after each token arrives. This is valuable in real-time applications, such as transcribing speech as it is spoken or processing log events as they stream in.

A bidirectional RNN must have the entire sequence before the backward pass can begin. There is no way to compute $\overleftarrow{h}_1$ until you have seen the last token. This introduces a latency of the full sequence length, making bidirectionality impractical for true streaming or online inference.

Workarounds exist but are costly. Chunk-based bidirectionality processes the sequence in overlapping windows, applying bidirectionality within each window but not across windows. This reduces latency at the cost of inter-chunk context. Another approach uses look-ahead buffers: wait for a fixed number of future tokens before processing the current token. This reduces but does not eliminate latency. None of these approximations match the quality of true full-sequence bidirectionality.

In temporal sequence prediction, including forecasting stock prices from past prices, predicting the next event in a clinical timeline, or modeling traffic patterns, conditioning on future information would constitute data leakage. Bidirectionality must be avoided or carefully scoped in such settings, since the backward pass sees information that would not be available at prediction time.

This is a subtle trap for practitioners. A model trained with bidirectionality on a temporal dataset will appear to perform excellently during training and validation if those splits do not carefully respect temporal ordering. But at deployment, the future tokens are unavailable, and the model's bidirectional representations are meaningless. The lesson is that architecture choices must reflect the inference-time information availability of the task.

Both the forward and backward passes of a BiRNN are sequential: each step depends on the previous step, so the computation for a sequence of length $T$ requires $T$ serial steps in each direction. This is the fundamental scalability limitation of all RNN architectures, bidirectional or not. Transformers overcome this with parallel self-attention that processes all positions simultaneously, which is one of the main reasons transformers scaled more efficiently to large models and large datasets than BiLSTMs.

For short sequences (up to a few hundred tokens), the sequential computation of BiLSTMs is rarely a bottleneck in practice: modern hardware can process them quickly. For long sequences (thousands of tokens), the sequential overhead becomes significant, and transformers have a clear advantage.

From a historical perspective, bidirectional RNNs represented a major advance in NLP. BiLSTM-CRF models dominated NER benchmarks for several years, and BiLSTMs formed the encoder component of state-of-the-art machine translation systems. But they have largely been superseded by transformer-based encoders, particularly BERT.

BERT uses self-attention rather than recurrence, which allows it to be bidirectional by design: every token attends to every other token directly in a single forward pass. The per-token representations produced by BERT are conceptually similar to BiLSTM outputs: each token embedding encodes full left and right context. But the attention mechanism achieves this more efficiently (no sequential computation) and with greater modeling capacity (every pair of tokens can interact directly, regardless of distance, without information bottlenecks in a hidden state). We will explore BERT and self-attention in detail later in the book.

This does not diminish the importance of understanding bidirectional RNNs. The conceptual insight, that encoding tasks benefit from full-context representations, carries directly from BiLSTMs to transformers. BERT's masked language modeling objective is specifically designed to force the model to use bidirectional context. The transition from BiLSTMs to BERT was an architectural evolution, not a conceptual revolution.

Bidirectional RNNs extend recurrent architectures to process sequences in both directions simultaneously. The key ideas are:

• Two passes: A forward network processes left to right; a backward network processes right to left. Both networks process the same input but have independent weight matrices.
• Concatenation: At each position, the forward and backward hidden states are concatenated, producing a representation that captures both left and right context.
• Full-context encoding: Every token's representation summarizes the entire surrounding sequence. This is the critical advantage over unidirectional RNNs for encoding tasks.
• Encoder-decoder split: Bidirectionality belongs in encoders, where the full input is available. Decoders must remain unidirectional to support autoregressive generation.
• Sequence labeling applications: BiRNNs excel at NER, POS tagging, chunking, semantic role labeling, and any task where per-token predictions depend on both local and global context. BiLSTM-CRF became the standard NER architecture before BERT.
• Sentence-level pooling: For tasks requiring a single sentence vector, strategies include concatenating final hidden states, mean pooling, max pooling, or attention-weighted pooling. Each makes different trade-offs between simplicity and expressiveness.
• Stacking and depth: Multiple stacked bidirectional layers build hierarchical representations, with lower layers capturing syntax and upper layers capturing semantics.
• ELMo and the transition to transformers: ELMo showed that pretrained deep BiLSTMs produce powerful contextual representations. BERT replaced the sequential BiLSTM with parallel self-attention, achieving joint bidirectionality rather than concatenated separate unidirectional LMs.
• Generation incompatibility: Bidirectional processing requires the full input sequence to be available in advance, making it incompatible with autoregressive generation. Language models must remain unidirectional.
• PyTorch usage: Setting bidirectional=True in nn.LSTM or nn.GRU enables bidirectionality. The output dimension becomes hidden_size * 2 because the two directional states are concatenated.

The next chapter examines stacked RNNs, where multiple recurrent layers are arranged vertically to build hierarchical sequence representations. The insights from bidirectionality carry forward: stacked bidirectional layers are a common pattern in high-performance sequence encoders.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about bidirectional RNNs.