Encoder-Decoder Architecture › Encoder-Decoder Architecture

The previous chapters explored encoder-only and decoder-only transformers as specialized tools: encoders for understanding, decoders for generation. But the original transformer from "Attention Is All You Need" was neither. It combined both components into a unified architecture designed for a specific challenge: transforming one sequence into another where input and output can have different lengths and vocabularies, as well as different structures.

Machine translation epitomizes this challenge. "The quick brown fox" has four words, but its German translation "Der schnelle braune Fuchs" also has four, while its Japanese translation might have a completely different structure. Summarization compresses a 1000-word article into a 50-word abstract. Question answering takes a question and a passage, then produces an answer span. These tasks share a common pattern: process an input sequence, understand it deeply, then generate an output sequence that relates to but differs from the input.

The encoder-decoder architecture handles this by dividing labor. The encoder builds rich, bidirectional representations of the input. The decoder generates the output autoregressively, attending both to its own previous outputs and to the encoder's representations through a mechanism called cross-attention. This chapter explores how these components interact, the mathematics of cross-attention, and when encoder-decoder models outperform their encoder-only or decoder-only alternatives.