Bidirectional RNNs › Visualizing the Architecture

Michael BrenndoerferMay 20, 20255 min read
Language AI HandbookAll chapters
Part 11 of 67Recurrent Neural Networks
Chapter 81Bidirectional RNNs › Visualizing the Architecture
Reading section 10 of 16 · Visualizing the Architecture · 5 minSection 10 of 16 · 5 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 81 of 513

Section 10 of 16 · 5 min read

Bidirectional RNNs › Visualizing the ArchitectureOne section · Sections in this chapter10 / 16 ↓
Sections in this chapter16
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

This section presents three visualizations: the per-position context coverage of a bidirectional pass, the parameter count tradeoff between unidirectional and bidirectional models at different hidden sizes, and the training loss curve from the POS tagger above.

Bidirectional Context Coverage

To make the information flow concrete, let's visualize how much each directional pass contributes to the representation at each position in a sample sentence.

Out[13]:
Visualization
Stacked bar chart showing forward and backward context contributions at each token position in a sentence.
Information flow in a bidirectional LSTM across a four-word sentence. The forward pass (blue) accumulates context from left to right, while the backward pass (orange) accumulates context from right to left. At each position, both streams are concatenated, giving every token a representation informed by the full sequence. The first token benefits most from the backward pass, which supplies all remaining right-context tokens.

The chart shows why position "Banks" (position 1) particularly benefits from the backward pass: the forward pass has seen only 1 token (itself), while the backward pass has seen all 4 tokens from right to left. After concatenation, the representation of "Banks" has access to the full 4-token sentence context.

Parameter Count Tradeoff

Doubling the output dimension by going bidirectional does not have to mean doubling the parameter count. If you halve the per-direction hidden size, the total stays comparable. The following chart shows total parameter counts for unidirectional and bidirectional LSTMs across a range of hidden sizes, using our toy vocabulary and tag set.

Out[14]:
Visualization
Line chart comparing parameter counts of unidirectional and bidirectional LSTMs across hidden sizes 16 to 256.
Total parameter counts for unidirectional versus bidirectional LSTM taggers across different hidden sizes. At a fixed hidden size, the BiLSTM has roughly twice the parameters of the unidirectional model. Setting the BiLSTM hidden size to half the unidirectional value (dashed line) yields nearly the same parameter count while providing full-context representations at every position.

The matched-capacity BiLSTM (dashed green) closely tracks the unidirectional model's parameter count while still providing full bidirectional context at each position. This is the typical trade-off practitioners use: to switch a unidirectional model to bidirectional without increasing total parameters, halve the hidden dimension.

Training Loss

Let's also look at how training progresses for our BiLSTM tagger.

Out[15]:
Visualization
Line chart showing training loss decreasing from above 1.5 to near zero over 100 epochs.
Training loss over 100 epochs for the BiLSTM POS tagger on a 6-sentence toy corpus. The loss decreases rapidly in the first 20 epochs and plateaus near zero, indicating the model has learned to tag all training sequences correctly. The vertical dashed line marks the epoch where loss first falls below 0.1.