Bidirectional RNNs › A Worked Example

Let's trace through a simple sentence to make the mechanics concrete.

Consider the four-word sentence: "Banks near the river"

We want to assign a POS tag to each word. The answer is NOUN, ADP (preposition), DET, NOUN.

Forward pass (left to right):

The forward LSTM starts with $\overrightarrow{h}_0 = \mathbf{0}$ and processes:

• Position 1, "Banks": $\overrightarrow{h}_1$ encodes "Banks" with no prior context. The representation is ambiguous because "banks" can be a noun (financial institutions, riverbanks) or a verb (to bank, to store).
• Position 2, "near": $\overrightarrow{h}_2$ encodes "near" given "Banks." The forward context "Banks near" strengthens the noun reading for "Banks" and establishes "near" as a preposition.
• Position 3, "the": $\overrightarrow{h}_3$ encodes "the" given "Banks near." The definite article after a preposition signals an upcoming noun phrase.
• Position 4, "river": $\overrightarrow{h}_4$ encodes "river" given the full left context. The right context is now complete.

Backward pass (right to left):

The backward LSTM starts with $\overleftarrow{h}_5 = \mathbf{0}$ (past the last token) and processes:

• Position 4, "river": $\overleftarrow{h}_4$ encodes "river" with no right context.
• Position 3, "the": $\overleftarrow{h}_3$ encodes "the river" from the right. The article-noun combination encodes cleanly.
• Position 2, "near": $\overleftarrow{h}_2$ encodes "near the river" from the right. The preposition-noun phrase combination establishes a locative prepositional phrase.
• Position 1, "Banks": $\overleftarrow{h}_1$ encodes "Banks" given "near the river" from the right, which is strong evidence for the noun reading. The locative context "near the river" does not follow verbs, only nouns.

Combined representations:

At each position, the forward and backward states are concatenated:

• "Banks": $[\overrightarrow{h}_1; \overleftarrow{h}_1]$ carries right context "near the river," providing strong evidence for NOUN
• "near": $[\overrightarrow{h}_2; \overleftarrow{h}_2]$ carries left context "Banks" and right context "the river," confirming ADP
• "the": $[\overrightarrow{h}_3; \overleftarrow{h}_3]$ carries both directions, confirming DET
• "river": $[\overrightarrow{h}_4; \overleftarrow{h}_4]$ carries left context "Banks near the," confirming NOUN

Each combined representation carries enough information to assign the correct POS tag with high confidence, even for the ambiguous "Banks" at position 1.

Notice that the ambiguity of "Banks" is resolved entirely by the backward context. If you removed the backward pass and gave the forward pass only, position 1 would have to make a decision based solely on the word "Banks" itself, with no surrounding evidence. The backward pass is what makes this case tractable.

Consider a longer and more realistic example where multiple tokens are ambiguous simultaneously: "The lead pipe in the kitchen was painted lead-colored."

This sentence contains two instances of "lead": the first as a noun (lead pipe, the heavy metal), and the second as an adjective (lead-colored). A POS tagger must assign NOUN to the first and ADJ to the second.

For the first "lead," the forward context at that position is just "The," which tells us nothing about whether "lead" is a noun or verb ("They lead the parade") or adjective. The backward context includes "pipe in the kitchen was painted lead-colored," which establishes a long noun phrase and makes the noun reading unambiguous.

For the second "lead," the forward context "...was painted" indicates an adjective or past participle is expected, and the backward context "colored" (in the compound adjective "lead-colored") confirms the adjectival reading.

Neither of these disambiguations is easy for a unidirectional model. Both are straightforward for a BiLSTM with full bidirectional context.