Bidirectional RNNs › Architecture: Forward and Backward Passes

A bidirectional RNN runs two independent recurrent networks over the same input sequence.

The forward network processes tokens in the standard left-to-right order:

$$\overrightarrow{h}_t = f(\overrightarrow{h}_{t-1}, x_t)$$

The backward network processes the same tokens in reverse order, from position $T$ down to position $1$:

$$\overleftarrow{h}_t = f(\overleftarrow{h}_{t+1}, x_t)$$

where:

• $\overrightarrow{h}_t$ is the forward hidden state at position $t$, summarizing context from the left (tokens $1$ through $t$)
• $\overleftarrow{h}_t$ is the backward hidden state at position $t$, summarizing context from the right (tokens $T$ down through $t$)
• $f$ is the same type of recurrent cell used in both directions, though the two networks have separate, independent parameters

At each position $t$, the two hidden states are concatenated to form a combined representation:

$$h_t = [\overrightarrow{h}_t ; \overleftarrow{h}_t]$$

where:

• $[\cdot ; \cdot]$ denotes vector concatenation
• If each directional hidden state has dimension $d$, the combined state has dimension $2d$

This combined representation at position $t$ encodes information from the entire sequence. The forward component carries context from tokens to the left of $t$, and the backward component carries context from tokens to the right. Every position benefits from full sentence-level context.

A subtle but important point is that the two directional networks have completely independent parameters. The forward LSTM has its own input-to-hidden weights, hidden-to-hidden weights, and biases. The backward LSTM has a separate, unshared set of weights. They are trained jointly via backpropagation, but gradient updates to one set do not directly affect the other.

This independence is intentional and beneficial. The forward network is optimized to summarize left context effectively. The backward network is optimized to summarize right context effectively. These tasks differ: left context tends to encode the topic and the syntactic head of phrases that came before, while right context tends to encode the consequences and completions of the current phrase. Separate parameters let each direction specialize in what it does best. Sharing parameters would force both directions to use the same representational strategy, which would likely hurt both.

Both directional networks are initialized independently, typically with the same random initialization scheme applied separately. During training, gradients flow backward through each direction independently: gradients for the forward LSTM flow from right to left through time (from position $T$ back to position $1$), and gradients for the backward LSTM flow from left to right through time (from position $1$ up to position $T$). Both sets of gradients are accumulated and applied to their respective parameters at each update step.

The combined representation at each position receives gradient contributions from both the output layer (through the classification head or whatever follows) and is passed back to both directional networks. This means the forward LSTM's weights are shaped by how well the combined forward-plus-backward representation classifies each position, not just by how well the forward representation alone would classify it. The two networks are coupled through the shared output loss, even though their parameters are independent.