Bidirectional RNNs › Bidirectional RNNs
Language AI HandbookAll chapters Bidirectional RNNs › Bidirectional RNNsOne section · Sections in this chapter1 / 16 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Every RNN architecture we have covered so far, from the vanilla RNN to the GRU, processes sequences in one direction: left to right. The network sees the first token, updates its hidden state, moves to the second token, and so on. By the time the model assigns a label to the fifth token in a sentence, it has seen tokens one through five, but nothing from tokens six onward.
This unidirectional constraint is a natural fit for language generation. When you are predicting the next word, you cannot look at future tokens because they have not been produced yet. But it creates a real problem for tasks where you already have the full sequence and your job is to analyze or label it. Consider named entity recognition: if you are deciding whether "Washington" in a sentence refers to a person or a city, the words that come after "Washington" often contain the decisive clue. "Washington visited the capital" and "Washington signed the treaty" present very different contexts, and the word "visited" or "signed" can only resolve the ambiguity if the model is allowed to look right as well as left.
Bidirectional RNNs solve this by running two separate passes over the sequence: one in the forward direction (left to right) and one in the backward direction (right to left). At each position, the representations from both passes are concatenated, giving every token a view of its entire surrounding context. This seemingly simple modification, introduced by Schuster and Paliwal in 1997, substantially improved sequence labeling performance and became the foundation for most pre-transformer NLP architectures.
The core insight is about the type of task you are solving. Some tasks require you to produce output before seeing all the input, and some tasks let you see the entire input before producing any output. For the second class of tasks, there is no reason to artificially limit the model to processing in one direction. Bidirectionality takes full advantage of the complete input, and the empirical gains are substantial.