Encoder-Decoder Framework › Visualizations

Michael BrenndoerferMay 15, 20254 min read
Language AI HandbookAll chapters
Part 12 of 67Sequence-to-Sequence
Chapter 83Encoder-Decoder Framework › Visualizations
Reading section 12 of 16 · Visualizations · 4 minSection 12 of 16 · 4 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 83 of 513

Section 12 of 16 · 4 min read

Encoder-Decoder Framework › VisualizationsOne section · Sections in this chapter12 / 16 ↓
Sections in this chapter16
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Out[16]:
Visualization
Line plot of cross-entropy training loss over epochs, showing rapid decrease then gradual convergence.
Training cross-entropy loss over 200 epochs for the seq2seq model on the word-reversal task. The loss drops steeply in early epochs as the model learns the encoder-decoder structure, then continues to decrease as the decoder refines its outputs. The smooth convergence reflects the stability provided by teacher forcing and gradient clipping.
Out[17]:
Visualization
Bar chart of BLEU scores for 20 training sentences, most bars near 1.0 with a mean dashed line.
Per-sentence BLEU scores for all 20 training examples after 200 epochs of training. Most sentences achieve BLEU scores near 1.0, indicating accurate character-level reproduction of the target sequences. The orange dashed line shows the mean BLEU across all sentences.
Out[18]:
Visualization
Line plot showing context dimensions per token declining as sequence length increases, for three hidden dim values.
Available context dimensions per source token as a function of sequence length, for three different hidden dimension sizes. As sequence length grows, each source token must share fewer and fewer dedicated dimensions in the fixed-size context vector. For a 100-token sequence with hidden dimension 128, there are fewer than 1.3 dimensions available per token, illustrating why the bottleneck becomes a practical barrier for long-sequence tasks.

The compression ratio plot shows concretely why the context vector bottleneck becomes severe for longer sequences. With a hidden dimension of 128, encoding a 5-token sentence gives about 25 dimensions per token. Encoding a 40-token sentence gives only about 3 dimensions per token. For a 100-token sentence, the model has fewer than 1.3 dimensions per token, which is almost certainly insufficient to represent all necessary information. This is precisely the regime where attention-based models gain their biggest advantages.

Out[19]:
Visualization
Grouped bar chart showing training stability increasing and inference alignment decreasing as teacher forcing ratio increases.
Conceptual tradeoff between training stability and inference alignment across different teacher forcing ratios. At ratio 1.0, every decoder input is a ground-truth token, giving a clean but optimistic training signal that creates exposure bias at inference. At ratio 0.0, the model trains on its own predictions, experiencing the same conditions as inference but converging more slowly. Mixed ratios around 0.5 to 0.75 offer a practical balance for most applications.

The teacher forcing tradeoff chart captures why the choice of forcing ratio matters. At ratio 1.0, training is highly stable because every decoder step receives correct context, but the model never learns to handle its own errors, creating a gap between training and deployment behavior. At ratio 0.0, the model practices on exactly the conditions it will face at inference, but training becomes very slow because early errors cascade through the sequence. Values around 0.5 to 0.75 offer a pragmatic balance for most applications.