Encoder-Decoder Framework › Summary

The encoder-decoder framework separates sequence-to-sequence transformation into two distinct phases. The encoder reads the source sequence token by token and compresses it into a fixed-size context vector, the final hidden state of the encoder RNN. The decoder then generates the target sequence autoregressively, starting from that context vector as its initial hidden state.

Key concepts covered in this chapter:

• The context vector is the sole communication channel between encoder and decoder. It must represent the entire source sequence in a fixed number of dimensions.
• The decoder uses the context vector as its initial state and generates tokens one at a time, conditioned on all previously generated tokens.
• Training uses cross-entropy loss with teacher forcing, where ground-truth previous tokens replace the decoder's own predictions to prevent error compounding during training.
• Bidirectional encoders process the source in both directions and consistently outperform unidirectional encoders by giving each position access to both left and right context.
• The fixed-size context vector creates a bottleneck that limits performance on long sequences, and the compression ratio worsens linearly as sequence length grows.
• BLEU score measures translation quality via n-gram precision with a brevity penalty. This provides an automatic and reproducible evaluation metric, though with known limitations for semantic evaluation.
• The framework generalizes beyond translation to summarization, code generation, image captioning, and any task that maps a variable-length source to a variable-length target.
• The seq2seq framework was the foundation for attention mechanisms and ultimately the transformer architecture, making it one of the most consequential architectural ideas in modern NLP.

The next chapter, Teacher Forcing, examines the training procedure in depth, including the exposure bias problem and scheduled sampling as a remedy. The Beam Search chapter then covers improved decoding strategies. Finally, Attention Intuition begins building up the mechanism that resolves the bottleneck problem at the heart of basic seq2seq models.