Encoder-Decoder Framework › Encoder-Decoder Framework

Every translation system faces the same fundamental challenge: the input and output are sequences of different lengths with no fixed correspondence between positions. When you translate "The cat sat on the mat" into French, the result "Le chat était assis sur le tapis" has a different number of tokens and a different word order. A standard feedforward network cannot handle this because it requires fixed-size inputs and produces fixed-size outputs. A single RNN cannot either, because it maps one sequence to a same-length sequence, one token at a time.

The encoder-decoder framework, introduced by Sutskever, Vinyals, and Le in 2014 and simultaneously by Cho et al., solved this problem by splitting the task into two stages. An encoder reads the entire input sequence and compresses it into a fixed-size summary called the context vector. A decoder then reads that summary and generates the output sequence one token at a time, autoregressively. This separation allows inputs and outputs to have completely different lengths, vocabularies, and even languages. It was the architectural breakthrough that brought neural machine translation to practical viability, and it became the foundation for summarization, code generation, question answering, and countless other sequence transformation tasks.

In this chapter, we build up the framework from first principles. We start with the conceptual split between encoding and decoding, work through the mathematics of the context vector, and examine the training procedure including teacher forcing. Then we implement a complete seq2seq model in PyTorch from scratch for a sequence transformation task, and we evaluate it using the BLEU score. We also discuss the central limitation of the fixed-size context vector, which motivates the attention mechanisms covered in later chapters of this part.