Bidirectional RNNs › Bidirectional LSTMs and GRUs

Nothing prevents you from using LSTM or GRU cells as the recurrent unit in each direction. In fact, bidirectional LSTMs (BiLSTMs) and bidirectional GRUs (BiGRUs) are far more common in practice than bidirectional vanilla RNNs, because they combine the long-range dependency handling of LSTM/GRU gates with the full-context encoding of bidirectionality.

A BiLSTM runs two independent LSTM networks over the sequence. Each direction maintains its own cell state $c_t$ and hidden state $h_t$:

Forward LSTM:

$$(\overrightarrow{h}_t, \overrightarrow{c}_t) = \text{LSTM}(\overrightarrow{h}_{t-1}, \overrightarrow{c}_{t-1}, x_t)$$

Backward LSTM:

$$(\overleftarrow{h}_t, \overleftarrow{c}_t) = \text{LSTM}(\overleftarrow{h}_{t+1}, \overleftarrow{c}_{t+1}, x_t)$$

The combined output is:

$$h_t = [\overrightarrow{h}_t ; \overleftarrow{h}_t]$$

where:

• $\overrightarrow{h}_t$ and $\overleftarrow{h}_t$ are the forward and backward LSTM hidden states at position $t$
• The cell states $\overrightarrow{c}_t$ and $\overleftarrow{c}_t$ are internal to each LSTM direction and are not concatenated for output

A BiGRU follows the same pattern with GRU cells instead of LSTM cells. Because GRUs have no separate cell state, only hidden states are concatenated. BiGRUs are slightly simpler and often competitive with BiLSTMs on many benchmarks while requiring fewer parameters.

The choice between BiLSTM and BiGRU typically comes down to the same considerations as the unidirectional case: dataset size, sequence length, and whether the full gating machinery of an LSTM is warranted given your compute budget.

Using LSTM or GRU cells in each direction amplifies the benefits of bidirectionality. A vanilla bidirectional RNN still suffers from vanishing gradients within each directional pass: the forward hidden state at position $T$ may contain very little information about position 1 if the sequence is long. This means that even though the representation at position 1 incorporates backward context, the backward context itself may be poorly encoded if the backward pass has trouble maintaining information over many steps.

LSTM cells address this precisely. Each directional pass in a BiLSTM can maintain information over long distances through its cell state, so the backward hidden state at position 1 contains well-preserved information from position $T$, even in sequences of hundreds of tokens. The combination of LSTM's temporal memory and bidirectionality's directional completeness is greater than either alone: you get good long-range memory in both directions simultaneously.

This is why BiLSTMs, rather than bidirectional vanilla RNNs, became the dominant architecture for NLP before the transformer era. Vanilla bidirectional RNNs are mostly of theoretical interest now.