Encoder-Decoder Framework › The Core Idea: Two Phases, One Bottleneck
Language AI HandbookAll chapters Encoder-Decoder Framework › The Core Idea: Two Phases, One BottleneckOne section · Sections in this chapter3 / 16 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The encoder-decoder framework is elegant in its simplicity. The encoder is responsible for understanding. It reads the input sequence one token at a time and produces a compressed representation of its meaning. The decoder is responsible for generating. It reads the compressed representation and produces the output sequence, one token at a time, where each new token depends on all previous tokens it has already generated.
The two components communicate through a single fixed-size vector, the context vector. This vector must capture everything the decoder needs to know about the input. It is the only information that flows from the input side to the output side.
The context vector is a fixed-size dense vector, typically the final hidden state of the encoder RNN, that summarizes the entire input sequence. It is the initial hidden state of the decoder, bridging the two components. Everything the decoder knows about the source comes from this single vector.
Both the encoder and decoder are typically recurrent neural networks, either vanilla RNNs, LSTMs, or GRUs. As we built up in Part XI, RNNs process sequences by maintaining a hidden state that is updated at each timestep. The encoder's hidden state evolves as it reads the input, and after the final token, that hidden state contains a compressed summary of the entire sequence. The decoder uses this summary as its starting point and generates tokens one by one until it produces a special end-of-sequence token.
The appeal of this design is that it imposes no length constraint on either side. The encoder processes however many tokens the source contains, and the decoder generates however many tokens the target requires. The two lengths are decoupled. The only constraint is that both sides share a common representation space of fixed dimensionality, the hidden dimension of the recurrent cells. This is the price of the decoupling: a single vector of fixed size must serve as the bridge, and the information-carrying capacity of that vector is bounded.
Why Not a Single RNN?
You might ask why we need two separate RNNs at all. Could we not just use a single RNN that takes in the source tokens and outputs the target tokens, like a many-to-many RNN? The problem is that a single RNN would need to simultaneously model two very different processes: reading and understanding source tokens, and generating target tokens. These require different behavior at each timestep.
More importantly, a naive many-to-many RNN would force the source and target sequences to have the same length, which is almost never the case in translation. The encoder-decoder architecture avoids this by first fully reading and compressing the source, then generating the target without being constrained to match the source length.
The price you pay is the bottleneck: the entire source sequence must be squeezed into a single fixed-size vector. This works acceptably for short sequences but becomes a serious limitation as sequences grow longer, because a vector of, say, 256 dimensions must somehow encode an entire paragraph of text. This limitation motivates the attention mechanism we will explore in the Attention Intuition and Bahdanau Attention chapters.