Encoder-Decoder Framework

Every translation system faces the same fundamental challenge: the input and output are sequences of different lengths with no fixed correspondence between positions. When you translate "The cat sat on the mat" into French, the result "Le chat était assis sur le tapis" has a different number of tokens and a different word order. A standard feedforward network cannot handle this because it requires fixed-size inputs and produces fixed-size outputs. A single RNN cannot either, because it maps one sequence to a same-length sequence, one token at a time.

The encoder-decoder framework, introduced by Sutskever, Vinyals, and Le in 2014 and simultaneously by Cho et al., solved this problem by splitting the task into two stages. An encoder reads the entire input sequence and compresses it into a fixed-size summary called the context vector. A decoder then reads that summary and generates the output sequence one token at a time, autoregressively. This separation allows inputs and outputs to have completely different lengths, vocabularies, and even languages. It was the architectural breakthrough that brought neural machine translation to practical viability, and it became the foundation for summarization, code generation, question answering, and countless other sequence transformation tasks.

In this chapter, we build up the framework from first principles. We start with the conceptual split between encoding and decoding, work through the mathematics of the context vector, and examine the training procedure including teacher forcing. Then we implement a complete seq2seq model in PyTorch from scratch for a sequence transformation task, and we evaluate it using the BLEU score. We also discuss the central limitation of the fixed-size context vector, which motivates the attention mechanisms covered in later chapters of this part.

To appreciate what the encoder-decoder framework achieved, you need to understand what came before it. For most of the 2000s and early 2010s, statistical machine translation (SMT) was the dominant paradigm. A typical SMT system was a carefully engineered pipeline consisting of three separate components: a language model that captured the fluency of the target language, a translation model that captured word-level and phrase-level correspondences between source and target, and a reordering model that handled the fact that word order changes across languages. These components were trained separately from different data sources and assembled into a decoder using log-linear models with hand-tuned weights.

The phrase-based SMT approach, as developed in systems like Moses, required enormous engineering effort. Building a high-quality system meant collecting millions of parallel sentence pairs, running word alignment algorithms like GIZA++ to establish token correspondences, extracting a phrase table, training an n-gram language model, and tuning the log-linear weights using minimum error rate training. Maintaining and extending these pipelines took teams of researchers years.

Early neural approaches tried to fit into this pipeline rather than replace it. Researchers trained neural networks to rescore n-best lists from SMT systems, or to supplement the phrase table with neural translation features. These hybrid systems showed modest improvements but required the full SMT infrastructure to remain in place.

The seq2seq papers of 2014 changed the calculus entirely. Sutskever, Vinyals, and Le showed that a single neural network, trained end-to-end, could approach the performance of carefully engineered SMT systems on the WMT English-French benchmark without any of the hand-crafted components. Cho et al. independently proposed a similar architecture and used it not to replace SMT but to enhance it by learning a continuous representation for phrases. This shows that neural encoder-decoders were a new kind of building block. Within two years, pure neural machine translation had surpassed SMT on every major benchmark, and Google replaced its production translation system in 2016 with an attention-based seq2seq model that served billions of users.

The historical leap combined a performance improvement with a conceptual change. The seq2seq framework showed that the right inductive bias for sequence-to-sequence tasks was not to hand-code alignment and reordering, but to learn a shared representation of meaning that could be decoded into any target form. This insight generalized far beyond translation, and it ultimately led to the transformer and the large language model paradigm.

The encoder-decoder framework is elegant in its simplicity. The encoder is responsible for understanding. It reads the input sequence one token at a time and produces a compressed representation of its meaning. The decoder is responsible for generating. It reads the compressed representation and produces the output sequence, one token at a time, where each new token depends on all previous tokens it has already generated.

The two components communicate through a single fixed-size vector, the context vector. This vector must capture everything the decoder needs to know about the input. It is the only information that flows from the input side to the output side.

Both the encoder and decoder are typically recurrent neural networks, either vanilla RNNs, LSTMs, or GRUs. As we built up in Part XI, RNNs process sequences by maintaining a hidden state that is updated at each timestep. The encoder's hidden state evolves as it reads the input, and after the final token, that hidden state contains a compressed summary of the entire sequence. The decoder uses this summary as its starting point and generates tokens one by one until it produces a special end-of-sequence token.

The appeal of this design is that it imposes no length constraint on either side. The encoder processes however many tokens the source contains, and the decoder generates however many tokens the target requires. The two lengths are decoupled. The only constraint is that both sides share a common representation space of fixed dimensionality, the hidden dimension of the recurrent cells. This is the price of the decoupling: a single vector of fixed size must serve as the bridge, and the information-carrying capacity of that vector is bounded.

You might ask why we need two separate RNNs at all. Could we not just use a single RNN that takes in the source tokens and outputs the target tokens, like a many-to-many RNN? The problem is that a single RNN would need to simultaneously model two very different processes: reading and understanding source tokens, and generating target tokens. These require different behavior at each timestep.

More importantly, a naive many-to-many RNN would force the source and target sequences to have the same length, which is almost never the case in translation. The encoder-decoder architecture avoids this by first fully reading and compressing the source, then generating the target without being constrained to match the source length.

The price you pay is the bottleneck: the entire source sequence must be squeezed into a single fixed-size vector. This works acceptably for short sequences but becomes a serious limitation as sequences grow longer, because a vector of, say, 256 dimensions must somehow encode an entire paragraph of text. This limitation motivates the attention mechanism we will explore in the Attention Intuition and Bahdanau Attention chapters.

The encoder is the "reader" half of the system. Its job is to process an arbitrary-length input and distill it into a compact representation that the decoder can work from. Before diving into the mathematics, it helps to build intuition about what this compression process looks like.

Think of the encoder as a reader who is given a passage and asked to write a summary on a notecard of fixed size. After reading each sentence, the reader updates their mental model of what they have read so far. By the end of the passage, the notecard should contain everything important about the passage, but compressed into whatever fits. Early sentences may not make it onto the notecard if they were less important, or they may be represented implicitly through their effect on the reader's evolving understanding. This is exactly what the LSTM encoder does: its hidden state at each step is an evolving compressed summary, and the final state is the notecard passed to the decoder.

The encoder processes the source sequence $x_1, x_2, \ldots, x_T$ and produces a sequence of hidden states $h_1, h_2, \ldots, h_T$. At each timestep $t$, the encoder computes a new hidden state based on the current input token and the previous hidden state:

where:

• $h_t \in \mathbb{R}^d$ is the encoder hidden state at time $t$
• $x_t$ is the embedding of the $t$-th input token
• $f_{\text{enc}}$ is the encoder recurrent cell (LSTM, GRU, or vanilla RNN)
• $d$ is the hidden size, a hyperparameter, typically 256 to 1024

After processing all $T$ input tokens, the context vector $c$ is taken as the final encoder hidden state:

where:

• $c \in \mathbb{R}^d$ is the context vector
• $h_T$ is the encoder hidden state after processing the last input token

For LSTM encoders, the encoder produces both a hidden state $h_T$ and a cell state $s_T$, and both are passed to the decoder. This pair $(h_T, s_T)$ together forms the context.

The encoder starts with a zero hidden state $h_0 = \mathbf{0}$ and accumulates information about the input as it reads token by token from left to right. By the time it reaches the end of the sequence, the hidden state theoretically contains a summary of everything the encoder has seen. In practice, vanilla RNNs struggle to retain information from the beginning of long sequences due to vanishing gradients, which is why LSTMs and GRUs are preferred for the encoder.

Before the encoder processes any tokens, the source vocabulary must be mapped to dense vectors using an embedding layer. If the source vocabulary has size $V_{\text{src}}$ and we use embedding dimension $d_e$, the embedding matrix $E_{\text{src}} \in \mathbb{R}^{V_{\text{src}} \times d_e}$ maps each integer token index to a continuous vector. These embeddings are learned jointly with the rest of the model during training.

The encoder receives these embedded vectors as input at each timestep, not the raw integer indices. The embedding layer is a required component: it performs dimensionality reduction from a one-hot space of size $V_{\text{src}}$ (which may be 30,000 or more tokens) to a dense space of size $d_e$ (typically 128 to 512), and it learns to position semantically related words close together in that space. A word like "run" and "sprint" will end up with similar embeddings after training on parallel corpora, because they appear in similar translation contexts.

Embedding dimension and hidden dimension do not need to match, though they often do in practice. If they differ, a linear projection is typically inserted between the embedding layer and the recurrent cell to map the embedding into the recurrent input space.

A standard encoder processes the input strictly left to right, meaning that the hidden state at position $t$ only incorporates information from tokens $x_1$ through $x_t$. The hidden state at position $T$ therefore has the most information, but the hidden state at position 1 has almost none about the rest of the sequence. When we use only the final state as the context vector, we are relying on the LSTM's ability to remember all relevant early tokens, which degrades with sequence length.

Bidirectional encoders address this by running two separate recurrent layers over the input: one in the forward direction (left to right) and one in the backward direction (right to left). The forward layer produces states $\overrightarrow{h}_1, \ldots, \overrightarrow{h}_T$ and the backward layer produces states $\overleftarrow{h}_1, \ldots, \overleftarrow{h}_T$. At each position, the two states are concatenated to give a full representation that incorporates context from both directions:

For the context vector in the basic encoder-decoder framework, we typically concatenate the final forward state and the initial (from the backward direction) backward state:

This gives the context vector access to the full forward pass summary and the full backward pass summary simultaneously. This provides a richer, more symmetric encoding of the source. In practice, bidirectional encoders consistently outperform unidirectional ones, especially for tasks where word order matters or where early tokens are important for understanding the sentence meaning.

The main cost of bidirectionality is that the context vector doubles in size (from $d$ to $2d$), which means the decoder must also use hidden states of size $2d$, or a linear projection is needed to bring the context back to size $d$ before feeding it to the decoder.

Just as we can stack LSTM layers in a standard RNN, we can stack them in the encoder. With $L$ layers, the first layer takes the embedded tokens as input, and each subsequent layer takes the hidden states of the layer below as input:

The final context vector is taken from the topmost layer: $c = h_T^{(L)}$. In the Sutskever et al. 2014 paper, a 4-layer LSTM encoder with reversed source input was used, and the depth was an important contributor to the model's performance. Deep encoders can learn hierarchical representations: lower layers might capture syntactic structure while higher layers capture semantic content, analogous to how convolutional networks build from edge detectors to object detectors.

Deeper encoders require correspondingly deeper decoders with matching layer counts, and the hidden state from each encoder layer is used to initialize the corresponding decoder layer. This ensures that information at different levels of abstraction is properly propagated across the bottleneck.

The decoder generates the target sequence $y_1, y_2, \ldots, y_{T'}$ one token at a time, conditioned on the context vector. It is an autoregressive model: each output token depends on all previously generated tokens.

At each decoder timestep $t$, the decoder computes a hidden state based on the previous decoder hidden state, the previous output token, and the context vector. The initial decoder hidden state is set to the context vector $c$.

where:

• $s_t \in \mathbb{R}^d$ is the decoder hidden state at time $t$
• $y_{t-1}$ is the embedding of the previous target token (or the start-of-sequence token at $t = 1$)
• $s_{t-1}$ is the previous decoder hidden state (with $s_0 = c$)
• $c$ is the context vector from the encoder
• $f_{\text{dec}}$ is the decoder recurrent cell

A linear projection followed by a softmax then converts the decoder hidden state $s_t$ into a probability distribution over the target vocabulary:

where:

• $W_o \in \mathbb{R}^{V_{\text{tgt}} \times d}$ is the output projection matrix
• $b_o \in \mathbb{R}^{V_{\text{tgt}}}$ is the output bias
• $V_{\text{tgt}}$ is the target vocabulary size
• $P(y_t \mid y_{< t}, x)$ is the probability of each possible next token given all previous tokens and the source

The decoder generates tokens by sampling or taking the argmax of this distribution at each step. It continues until it produces the special end-of-sequence token <EOS>, after which generation stops.

The decoder generation process requires special tokens to signal the beginning and end of the sequence. A start-of-sequence token <SOS> (sometimes called <BOS>) is fed as the first decoder input at $t = 1$. When the decoder generates the end-of-sequence token <EOS>, generation terminates. During training, target sequences are wrapped with these tokens: the decoder input target sequence starts with <SOS> and the target labels end with <EOS>.

These special tokens play a specific semantic role. The <SOS> token is both a signal to begin generating and the initial "seed" that the decoder uses, in combination with the context vector, to choose the first target token. The decoder's representation of <SOS> in embedding space is learned during training, and it effectively encodes the concept of "beginning of sequence in this language with this context." Without a start token, the decoder has no way to make its first prediction, since its autoregressive design requires a previous token at every step.

The <EOS> token serves a similarly critical function. It gives the decoder a way to signal "I am done" rather than generating forever. At inference time, the generation loop checks for this token at every step and stops when it is produced. During training, the model is supervised to produce <EOS> at the correct position in the target sequence, so it learns to generate sequences of the right length.

In the most straightforward implementation, the context vector is used once: as the initial hidden state of the decoder. After the first decoder step, the context information is only accessible through the decoder's evolving hidden state, which must carry it forward across all decoding steps.

An alternative is to concatenate the context vector to the decoder's input at every step. Under this scheme, the decoder at each position sees both the embedding of the previous token and the full context vector:

This gives the decoder a direct, unmediated connection to the context at each step, preventing the context information from degrading as the decoder hidden state evolves over many timesteps. Empirically, this variant tends to perform better on longer sequences, since the decoder does not have to rely solely on memory to retain the source summary.

A third approach concatenates the context vector to the output projection input, giving the vocabulary distribution direct access to the source encoding at every position. In practice, many implementations combine these ideas, concatenating context to both the input and the output projection.

At inference time, the simplest strategy is greedy decoding: at each step, the decoder selects the single highest-probability token and feeds it as input for the next step. This runs in $O(T')$ forward passes and is fast, but it is not guaranteed to find the globally most probable sequence.

The problem is that the greedy choice at step $t$ may close off more probable paths at steps $t+1, t+2, \ldots$. Consider a translation where the correct first word is "unfortunately," which might have probability 0.30 at step 1, but the second most probable word "sadly" has probability 0.25. Greedy decoding picks "unfortunately," but if "sadly" leads to a more probable continuation overall, greedy decoding would have made a suboptimal local decision.

Beam search addresses this by keeping the top $k$ candidate partial sequences at each step and expanding all of them before pruning back to the top $k$. This allows the decoder to explore multiple alternatives simultaneously, catching cases where a lower-probability early token leads to a higher-probability full sequence. The Beam Search chapter covers this algorithm in detail. For our implementation in this chapter, we use greedy decoding for simplicity and speed.

The context vector is simultaneously the architecture's greatest strength and its most significant weakness. It enables encoder-decoder architectures to handle variable-length inputs and outputs without any fixed correspondence between positions. But it forces the encoder to compress arbitrarily long sequences into a single fixed-size vector.

Consider what this means for machine translation of long documents. A 500-word source text must be compressed into, say, a 512-dimensional vector. All syntactic structure, all semantic content, all discourse relationships must fit into those 512 numbers. The decoder must then reconstruct a coherent, accurate translation of all 500 words from this single vector. The longer the source sequence, the harder this compression becomes.

The empirical evidence for this degradation is clear: seq2seq models without attention perform well on short sentences (under 20 tokens) but their BLEU scores degrade noticeably as sentence length increases. The encoder's hidden state at position $T$ has strong memory of tokens near the end of the sequence, but weak memory of tokens near the beginning, due to the way gradients vanish over time in RNNs.

This bottleneck motivated the development of attention mechanisms. Rather than forcing all information through a single vector, attention allows the decoder at each step to selectively look back at all encoder hidden states $h_1, h_2, \ldots, h_T$ and form a weighted combination that emphasizes the most relevant source positions. We will build up attention from this motivation in the subsequent chapters of this part.

From an information-theoretic perspective, the bottleneck problem is easy to quantify. A hidden state vector of dimension $d$ with floating-point precision can, in principle, carry at most $O(d)$ independent bits of information, ignoring numerical precision details. For a source sequence of $T$ tokens, each drawn from a vocabulary of size $V$, the raw information content is $O(T \log V)$ bits. The ratio between the information that must be transmitted and the channel capacity is:

For a sentence of 20 tokens from a vocabulary of 30,000 words, and a hidden dimension of 512, this ratio is approximately:

This is manageable: there are about 0.58 bits of raw information per dimension, leaving some headroom. But for a sentence of 100 tokens, the ratio rises to approximately 2.9, and the raw vocabulary information alone exceeds the capacity of the context vector. In practice, not all of this information needs to be retained verbatim (grammar can be compressed heavily, and much semantic content is redundant), but the trend is clear: as sequences grow, the bottleneck becomes increasingly severe.

This information-theoretic perspective also explains why neural seq2seq models improved so dramatically when attention was added. Attention does not eliminate the fixed-size hidden state; each encoder hidden state $h_t$ is still the same size $d$. What attention does is increase the total information bandwidth available to the decoder at each step. Instead of reading from a single $d$-dimensional vector, the attention decoder reads from a dynamically weighted combination of all $T$ encoder hidden states, giving it access to up to $T \times d$ dimensions of information (though the weighted combination reduces this to $d$ again). The weighting is content-dependent: the decoder can retrieve whichever source positions are most relevant for generating the current target token.

The beauty of the encoder-decoder framework is that both components are trained jointly as a single model through standard backpropagation. There is no separate pre-training of the encoder or decoder; the entire model learns simultaneously to encode source sequences well and to decode those encodings into target sequences accurately.

This end-to-end training is one of the most important properties of the framework. In the earlier SMT era, the alignment model, the translation model, and the language model were trained separately on different objectives and then combined. Each component was optimized for a proxy objective that approximated the true goal (producing high-quality translations) but was not the actual goal. End-to-end training aligns every parameter in both the encoder and decoder directly with the translation objective, allowing the model to find internal representations that jointly optimize understanding and generation.

Training uses standard cross-entropy loss. Given a source sequence $x$ and a gold target sequence $y_1^*, y_2^*, \ldots, y_{T'}^*$, the model produces a probability distribution over the vocabulary at each target position. The loss is the negative log-likelihood of the correct tokens:

where:

• $T'$ is the length of the target sequence
• $y_t^*$ is the correct target token at position $t$
• $P(y_t^* \mid y_{< t}^*, x)$ is the model's probability assigned to the correct token

This is the standard cross-entropy loss averaged over positions, the same objective used for language model training. The model is trained to maximize the probability of the correct target sequence given the source.

One subtlety is that this loss decomposes as a product of conditional probabilities. The joint probability of the full target sequence is:

Taking the log converts this product to a sum, which is the form we minimize. This factorization assumes that each target token, given the source and all previous correct target tokens, can be predicted as if it were an independent classification problem. This is exactly what teacher forcing enables during training.

During training, the decoder has two choices for what to use as the previous token input $y_{t-1}$. It could use its own prediction from the previous step, or it could use the ground-truth token. Using the model's own predictions is called autoregressive training; using the ground-truth tokens is called teacher forcing.

Teacher forcing is almost universally used during training because it dramatically accelerates convergence. If the model makes a mistake at position 3, that error would propagate through positions 4, 5, 6, etc. when using autoregressive training, causing a cascade of compounding errors. With teacher forcing, each position is trained independently on correct context, making the training signal much cleaner.

The downside is an exposure bias: the model never sees its own mistakes during training, so it can behave poorly when deployed, because it must now rely on its own (potentially incorrect) previous predictions. This training-inference mismatch is a known issue, and various remedies like scheduled sampling exist, where you gradually replace ground-truth tokens with model predictions as training progresses. The Teacher Forcing chapter covers these alternatives in depth.

In practice, training batches contain sequences of different lengths. To pack them into matrices for efficient GPU computation, shorter sequences are padded to the length of the longest sequence in the batch using a special <PAD> token. The loss is then masked so that padded positions do not contribute to the gradient. This ensures that padding does not confuse the model or artificially inflate the loss.

The masking step is important: without it, the model would be penalized for generating incorrect tokens at pad positions, which would distort the gradient and slow learning. In PyTorch, nn.CrossEntropyLoss accepts an ignore_index argument that automatically zeros out the loss at any position where the target label equals the specified index, making masking straightforward.

Pack-and-sort is another efficiency trick used in production training pipelines. Batches are sorted by source sequence length, and sequences within a batch are packed with minimal padding. The torch.nn.utils.rnn.pack_padded_sequence and pad_packed_sequence utilities support this workflow, allowing the LSTM to skip over padded positions entirely rather than processing them. This can save substantial compute for datasets with high variance in sequence length.

Training seq2seq models requires careful attention to optimization settings. A few considerations stand out.

First, gradient clipping is essential. Recurrent networks are prone to gradient explosions, where gradients grow exponentially as they propagate through many timesteps. Clipping the gradient norm to a maximum value (typically 1.0 or 5.0) prevents these instabilities. The Sutskever et al. paper used gradient clipping with a threshold of 5.0, and without it, training diverged.

Second, learning rate scheduling matters. Starting with a moderately high learning rate (around $1 \times 10^{-3}$ for Adam or $1.0$ for SGD with gradient clipping) and decaying it when validation loss stops improving is a common strategy. Transformer-based models introduced more sophisticated warmup schedules, but for RNN-based seq2seq, simple step decay or reduce-on-plateau typically suffices.

Third, initialization affects convergence. The Sutskever et al. 2014 model used uniform initialization in the range $[-0.08, 0.08]$ for all parameters and found this important for stable training. Modern frameworks use Xavier or He initialization by default, which is generally adequate.

Fourth, input sequence reversal is a somewhat surprising trick from the Sutskever et al. paper. They reversed the source sequence before feeding it into the encoder and observed substantial BLEU improvements. The intuition is that reversing brings the first source token closer to the first target token in the unrolled sequence of encoder-decoder steps, strengthening the gradient path between the two and making it easier for the decoder to learn to generate the beginning of the target sequence. The decoder must generate the target left to right, and with a reversed source, the last token the encoder sees is the first source token, which often corresponds to the first target token in translation. This shortens the path through the bottleneck for the tokens that matter most.

The encoder-decoder framework generalizes well beyond machine translation. Any task that maps one variable-length sequence to another is a natural fit.

Translation was the motivating application and remains the canonical benchmark. Given a source sentence in one language, the decoder generates a sentence in the target language. The model must implicitly learn the vocabulary, grammar, and semantics of both languages from parallel training corpora. What makes this particularly remarkable is that the model has no explicit alignment table or phrase dictionary: it learns to align source and target concepts implicitly through the continuous representations in the context vector and, once attention is added, through the attention weights.

One practical challenge is vocabulary coverage. Real translation requires handling rare words, proper nouns, technical terms, and code-switching between languages. Early seq2seq models used fixed vocabularies of around 30,000 to 80,000 tokens, and words outside this vocabulary were mapped to a single <UNK> (unknown) token. This severely limited quality on any sentence with rare terminology. The development of subword tokenization, and later byte-pair encoding (as covered in the Subword Tokenization part of this book), addressed the vocabulary problem by breaking rare words into frequent subword units that the model can handle.

In summarization, the source is a document and the target is a shorter summary. Abstractive summarization uses an encoder-decoder model where the decoder generates new summary text rather than extracting source sentences verbatim. The source is typically much longer than the target, pushing the context vector bottleneck to its limits, which is why attention-based models dramatically outperform basic seq2seq on this task.

An additional challenge in summarization is that the source document may contain specific entities, numbers, and named facts that must appear verbatim in the summary. The copy mechanism, which allows the decoder to directly copy tokens from the source rather than generating them from the vocabulary, was developed specifically to address this need. The copy mechanism can be viewed as a special attention pattern that, instead of building a weighted average of encoder hidden states, selects a source position directly and outputs the token at that position.

Code generation maps a natural language description (the source) to code (the target). This is another natural seq2seq task, though the output vocabulary and structure differ significantly from natural language: code has strict syntactic rules and the decoder must track things like indentation, brackets, and variable names across long spans.

The deterministic structure of programming languages creates both opportunities and challenges. On one hand, it is easier to evaluate correctness objectively (does the code run and produce the right output?) than to evaluate translation quality. On the other hand, a single misplaced bracket or wrong indentation level produces code that fails entirely, making precision more important than in natural language generation where small errors are often tolerable.

Seq2seq models have been applied to question answering (question as source, answer as target), dialogue systems (user utterance as source, response as target), speech recognition (acoustic features as source, text as target), and image captioning (image features as source, caption as target). The framework is remarkably versatile.

For image captioning, the encoder is replaced with a convolutional neural network that produces a fixed-size feature vector from the image, and this feature vector plays the role of the context vector that initializes the decoder. The decoder then generates a natural language caption. The "source" in this case is not a sequence at all, but the encoder-decoder interface remains the same: compress the input into a fixed representation, then decode into a sequence. This flexibility shows that the framework's core idea, separating representation from generation, is not specific to sequence inputs.

Let us trace through a small example to make the mechanics concrete. Suppose we want to learn a mapping from phrases to their word-reversed versions: "the cat sat" maps to "sat cat the".

Encoding phase:

The source tokens are ["<SOS>", "the", "cat", "sat", "<EOS>"]. The encoder embedding layer maps each token index to a 64-dimensional vector. The LSTM encoder processes these embeddings left to right, updating its hidden state and cell state at each step.

After reading <SOS>, the hidden state $h_1$ contains information about the start of a sequence, which is fairly generic. After reading "the", $h_2$ begins to reflect that the sequence starts with a common function word. After "cat", $h_3$ knows about "the cat," a likely subject phrase. After "sat", $h_4$ reflects a complete verb phrase "the cat sat." After <EOS>, $h_5$ contains the full context of a short sentence describing a sitting cat.

The encoder produces hidden states $h_1, h_2, h_3, h_4, h_5$. After processing the final <EOS> token, we take $c = (h_5, s_5)$ as the context vector pair.

Decoding phase:

The decoder starts with hidden state $s_0 = c$ and first input $y_0 = \text{embedding}(\text{<SOS>})$. At each step:

• $t = 1$: The decoder computes $s_1 = \text{LSTM}(y_0, s_0)$, projects to logits, and predicts "sat" as the highest-probability token.
• $t = 2$: Feed the embedding of "sat", compute $s_2$, and predict "cat".
• $t = 3$: Feed the embedding of "cat", predict "the".
• $t = 4$: Feed the embedding of "the", predict <EOS>.

The final output is "sat cat the", which is the correct word-reversed version.

During training with teacher forcing, the decoder input at $t = 2$ would be the correct token "cat" regardless of what the decoder predicted at $t = 1$.

What the context vector must encode:

For this word-reversal task, the context vector must capture the complete ordered sequence of content words in the source, so the decoder can emit them in reverse order. This is a particularly demanding test of the context vector because it requires retaining precise positional information about every word. In real translation, the context vector does not need to store verbatim source content; it needs to store the meaning, which is a more compressed representation. Even so, for real translation of moderately long sentences, the context vector quickly becomes the limiting factor.

Let us now build a complete seq2seq model from scratch using PyTorch. We will implement it for a character-level sequence transformation task (mapping phrases to their word-reversed versions) to keep the vocabulary small and training fast.

We start by importing all required libraries and preparing training data.

The training data maps English phrases to their word-reversed versions, which is a minimal but illustrative sequence transformation task. With 29 vocabulary items (26 letters plus space and special tokens), the model can focus on learning the seq2seq mechanics rather than a large vocabulary. Note that the character-level encoding means the model treats each individual character as a token, not each word, so "the cat sat on the mat" becomes a sequence of 22 character tokens plus the start and end tokens.

The encoder wraps an embedding layer and an LSTM. It reads all source tokens and returns the final hidden and cell states.

The forward method returns the LSTM's final (hidden, cell) tuple, which becomes the context vector. We discard the intermediate hidden states outputs because the basic seq2seq model uses only the final state. The attention-based models in later chapters will use all intermediate states: the attention mechanism needs access to $h_1, h_2, \ldots, h_T$ to form the context-sensitive weighted sum at each decoding step.

The padding_idx argument to nn.Embedding sets the embedding vector for the pad token to all zeros and keeps its gradient at zero during training. This ensures that padded positions do not affect the learned embeddings for real tokens.

The decoder also wraps an embedding layer and an LSTM, but it generates one token at a time and uses the context vector as its initial hidden state.

The decoder processes one token at a time. It takes a single token input_token, the current hidden state hidden, and cell state cell, and produces logits over the vocabulary plus updated states. This single-step design lets us implement teacher forcing easily by controlling which token we feed in at each step.

The fc_out layer is a linear projection from the hidden dimension to the full vocabulary size. At each step, the decoder's hidden state encodes both what has been generated so far and what the encoder told it about the source. The linear projection maps this rich representation to an unnormalized score for every possible next token, and the softmax during training (handled internally by nn.CrossEntropyLoss) converts these to probabilities.

The seq2seq wrapper combines the encoder and decoder and implements teacher forcing.

The teacher_forcing_ratio controls the mix: at 1.0 we always use ground truth (pure teacher forcing), at 0.0 we always use the model's own predictions (fully autoregressive), and values in between give a blend. Notice that the loop starts at $t = 1$, not $t = 0$. Position 0 in the target is the <SOS> token, which is the initial decoder input but not a target label. The output at position $t$ is the model's prediction for target position $t$, so we align prediction at $t$ with label at $t$.

This forward pass accumulates the full output tensor at each decoding step, which is what we need for computing the loss during training. At inference time, we do not need to precompute the full output tensor; we can greedily pick the best token at each step and stop when <EOS> is generated.

The loss drops substantially from its initial value toward near zero, confirming the model has learned to reproduce the training sequences. With only 20 training examples and teacher forcing enabled throughout, the model memorizes the training set quickly. The ignore_index=PAD_IDX argument to CrossEntropyLoss ensures that padded positions do not contribute to the gradient, while clip_grad_norm_ prevents gradient explosions that are common in LSTM training.

The reshape operations before computing the loss are important. The model outputs a tensor of shape (batch, tgt_len, vocab_size), but CrossEntropyLoss expects predictions of shape (N, vocab_size) and targets of shape (N,). Slicing off position 0 (which is the SOS token, not a target label) and then flattening aligns the prediction for step $t$ with the target label at position $t$.

At inference time, there is no teacher forcing. The decoder uses its own predicted token at each step.

The model accurately reproduces the word-reversal transformation for the training sentences. This demonstrates that the encoder successfully compressed the source phrase into a context vector containing enough information for the decoder to produce the correct output. Notice that the inference function uses model.eval() and torch.no_grad(), which disables dropout and gradient computation respectively. In eval mode, dropout layers pass their inputs unchanged, and gradient tracking is disabled to save memory and computation.

The key hyperparameters for the seq2seq model are:

• embed_dim: Dimensionality of token embeddings. Larger values capture richer representations but increase memory and computation.
• hidden_dim: Dimensionality of the LSTM hidden state. This is the size of the context vector, the fundamental bottleneck of the architecture.
• n_layers: Number of stacked LSTM layers. Multiple layers allow hierarchical representation, as covered in the Stacked RNNs chapter.
• teacher_forcing_ratio: Probability of using ground-truth tokens during training. Higher values speed convergence but increase exposure bias.
• dropout: Regularization for the embedding and LSTM layers. Helps prevent overfitting on larger datasets.

Evaluating sequence generation is harder than evaluating classification. You cannot just compare logits to labels because there are multiple valid translations of any source sentence. The Bilingual Evaluation Understudy (BLEU) score is the most widely used automatic metric for machine translation.

Before BLEU was proposed by Papineni et al. in 2002, evaluating machine translation required human judges to rate translation quality on a scale, an expensive and slow process. Progress in the field was measured by periodic evaluation campaigns that could take months to organize. BLEU made it possible to evaluate a system in seconds, allowing rapid iteration and system comparison. The impact on research productivity was clear: researchers could now run ablation studies and hyperparameter sweeps with automatic evaluation, dramatically accelerating progress.

The core challenge BLEU addresses is that translation quality cannot be measured by exact string match. "The cat sat on the mat" and "A cat was sitting on the mat" might both be valid translations of the same French sentence, but exact match would give the second translation a score of zero against the first as a reference. BLEU instead measures n-gram overlap, which captures the intuition that good translations share many words and phrases with human reference translations, even if the exact wording differs.

BLEU measures the n-gram overlap between the generated translation and one or more reference translations. It computes the modified precision for n-grams of orders 1 through 4, then combines them geometrically and applies a brevity penalty to discourage overly short translations.

The modified n-gram precision $p_n$ captures how many of the candidate's n-grams appear in the reference, clipped so the model cannot cheat by repeating common words. Formally:

where:

• The numerator counts how many candidate n-grams appear in the references, capped by the maximum count in any single reference
• The denominator counts all n-grams in the candidate translation
• The clipping prevents the model from inflating its score by repeating high-frequency words

The brevity penalty $BP$ discourages short outputs:

where:

• $c$ is the total length of all candidate translations
• $r$ is the reference length that best matches each candidate

The final BLEU score combines these components:

where:

• $N = 4$ uses n-grams of orders 1 through 4
• $w_n = 1/N = 0.25$ gives uniform weights across n-gram orders
• The exponential of the weighted sum equals the geometric mean of the individual precisions

BLEU scores range from 0 to 1. In practice:

• BLEU above 0.6 indicates excellent, near-human quality
• BLEU between 0.4 and 0.6 indicates good, understandable translations
• BLEU between 0.2 and 0.4 indicates decent but significant errors
• BLEU below 0.2 indicates poor quality

While BLEU is practical and widely used, it has well-known limitations worth understanding. First, BLEU is not a good metric for individual sentences: it was designed for corpus-level evaluation where it averages over thousands of examples, and single-sentence BLEU scores are noisy and unreliable. Second, BLEU can reward systems that match the surface form of a reference without capturing the meaning, and it can penalize paraphrases that are semantically equivalent but lexically different. Third, BLEU does not measure fluency directly: a translation that scrambles the order of correct n-grams might get a non-trivial unigram BLEU score even though the output is barely readable.

More recent metrics like BERTScore and COMET attempt to address these limitations by measuring semantic similarity using contextual embeddings rather than surface n-gram overlap. These metrics correlate better with human judgments but are more expensive to compute. For the purposes of this chapter, BLEU is the appropriate metric because it is interpretable and directly connected to the n-gram overlap intuition.

The BLEU score on the training set reflects how well the model has learned the training sequences. A mean BLEU near 1.0 indicates that the decoder is accurately reproducing the correct reversed-word sequences, confirming the model has captured the source-to-target transformation.

The teacher forcing tradeoff chart captures why the choice of forcing ratio matters. At ratio 1.0, training is highly stable because every decoder step receives correct context, but the model never learns to handle its own errors, creating a gap between training and deployment behavior. At ratio 0.0, the model practices on exactly the conditions it will face at inference, but training becomes very slow because early errors cascade through the sequence. Values around 0.5 to 0.75 offer a pragmatic balance for most applications.

Our implementation above is simplified to aid learning. Real production-scale seq2seq systems for machine translation involve several additional components and considerations.

A toy model trained on 20 sentences illustrates the mechanics but cannot generalize. Real translation models require millions to hundreds of millions of parallel sentence pairs. The WMT (Workshop on Machine Translation) datasets for English-French, English-German, and English-Czech are standard benchmarks, containing 40 million, 4.5 million, and 1 million pairs respectively. Collecting and cleaning such datasets is a major engineering effort: web-scraped parallel data from multilingual websites, Europarl transcripts, UN documents, and translated news articles are all used.

Data quality matters as much as quantity. Noisy pairs, where the source and target are not translations of each other, degrade model quality. Deduplication, length-ratio filtering, and language identification are standard preprocessing steps that remove the most egregiously misaligned pairs.

Word-level vocabularies create the out-of-vocabulary problem: words that do not appear in training are mapped to <UNK>, losing their meaning entirely. Character-level models as in our implementation avoid this but must model very long sequences (a single word may span multiple timesteps), which strains the context vector.

Byte-pair encoding (BPE) and its variants solve this by learning a subword vocabulary of 30,000 to 64,000 pieces, where common words appear as single tokens and rare words are split into common subword pieces. This allows the model to handle any word in any language by composing subword pieces, while keeping sequence lengths manageable. We covered BPE in depth in the Subword Tokenization part of this book. For translation, both the source and target vocabularies may be trained jointly on the combined parallel corpus, creating a shared subword vocabulary that facilitates transfer between the two languages.

Greedy decoding is convenient but leaves quality on the table. Beam search with a beam width of 4 to 10 typically improves BLEU by 2 to 5 points compared to greedy decoding on standard benchmarks. The computational cost scales linearly with beam width, since you are maintaining $k$ parallel decoder states instead of one. For real-time translation systems, beam width is a tunable latency-quality tradeoff: wider beams are better but slower.

Beam search also introduces a length normalization issue. Because beam search multiplies probabilities across timesteps, longer sequences accumulate more probability terms and tend to have lower total probability than shorter sequences, causing the model to prefer unnaturally short translations. Dividing the sequence log-probability by the sequence length (possibly raised to a power less than 1 to soften the normalization) counteracts this length bias.

Once you have a working seq2seq framework, extending it to multiple language pairs is relatively straightforward. The Google Multilingual Neural Machine Translation (MNMT) system demonstrated that a single seq2seq model could translate between dozens of language pairs simultaneously, simply by prepending a target language tag to the source sequence (for example, <2fr> to indicate "translate to French"). This massively multilingual approach allows the model to transfer representations across related languages, improving translation quality for low-resource language pairs that benefit from cross-lingual sharing.

The encoder-decoder framework was a landmark contribution, but it comes with several well-understood limitations that drove follow-on research.

The context vector bottleneck is the most fundamental limitation. A fixed-size vector must encode everything from a variable-length source, and the model has no mechanism to look back at specific source positions when generating a particular target token. When translating a long sentence, the decoder must rely entirely on the context vector, which may have forgotten early parts of the source by the time the encoder finishes reading.

Empirically, BLEU scores for basic seq2seq models drop noticeably on sentences longer than 20 to 30 tokens. The encoder's hidden state at position $T$ has strong memory of tokens near the end of the sequence but weak memory of tokens near the beginning, due to the way gradients vanish over time in RNNs. The attention mechanism, introduced by Bahdanau et al. in 2015, directly addresses this by giving the decoder soft access to all encoder hidden states at every decoding step.

The magnitude of the improvement from attention is substantial. On the WMT English-French benchmark, adding attention to the basic seq2seq model improved BLEU by roughly 3 to 5 points, closing much of the gap between the basic neural model and the then-best phrase-based systems. More importantly, the attention-equipped model's performance no longer degraded sharply with sentence length: it maintained near-constant BLEU across the full range of sentence lengths, while the attention-less model deteriorated rapidly beyond 20 tokens.

Teacher forcing creates a training-inference mismatch. During training, the decoder always receives correct context, the ground-truth previous tokens. During inference, it receives its own predictions, which may be wrong. Errors compound: a mistake at position 5 creates wrong context for positions 6, 7, 8, and so on. This exposure bias becomes especially harmful in long sequences.

Scheduled sampling partially mitigates this, at the cost of more complex training dynamics. The Teacher Forcing chapter explores these tradeoffs in detail.

Basic seq2seq uses greedy decoding at inference: the decoder always picks the highest-probability token at each step. This is fast but suboptimal because a locally optimal sequence of token choices does not necessarily maximize the total sequence probability. Beam search maintains multiple candidate hypotheses simultaneously and selects the globally best sequence, achieving higher BLEU scores at increased computational cost. The Beam Search chapter covers this.

Because the LSTM encoder processes tokens one at a time, left to right, the encoder computation cannot be parallelized across the sequence. Each hidden state depends on the previous one. For long sequences or large models, this sequential dependency creates a bottleneck in information capacity and in wall-clock training time. The transformer architecture addresses this by replacing recurrence with self-attention, which can process all positions in parallel. As we will see in later chapters, this parallelism was arguably the most important practical advantage of transformers over seq2seq, enabling training on orders of magnitude more data.

Despite these limitations, the encoder-decoder framework changed neural sequence modeling. Before seq2seq, neural machine translation systems required careful alignment models, phrase tables, and language models assembled by hand. The seq2seq model replaced all of that with a single, end-to-end trained neural network. The 2014 papers by Sutskever et al. and Cho et al. demonstrated competitive or superior performance on English-French translation compared to state-of-the-art phrase-based systems, purely from parallel sentence data.

The framework also opened up a vast range of new applications. Summarization, dialogue, code generation, image captioning, speech recognition, and question answering all became tractable with the same basic architecture. The combination of encoder-decoder with attention became the standard for all sequence transformation tasks, and the attention mechanism within this framework eventually inspired the transformer architecture that dominates NLP today. The original transformer paper, "Attention is All You Need," replaced the recurrent encoder and decoder with stacked self-attention layers, but kept the fundamental encoder-decoder split intact. The encoder still produces a sequence of representations from the source, and the decoder still generates the target autoregressively, consulting the encoder outputs through cross-attention at each step.

Virtually every modern language model is a descendant of the encoder-decoder insight. BERT is an encoder trained on masked language modeling. GPT is a decoder trained on causal language modeling. T5 and BART are full encoder-decoder models. The specific architectures have evolved enormously, but the underlying intuition, that reading and generating are different tasks that benefit from different architectures connected through a learned representation, remains at the heart of the field.

The encoder-decoder framework separates sequence-to-sequence transformation into two distinct phases. The encoder reads the source sequence token by token and compresses it into a fixed-size context vector, the final hidden state of the encoder RNN. The decoder then generates the target sequence autoregressively, starting from that context vector as its initial hidden state.

Key concepts covered in this chapter:

• The context vector is the sole communication channel between encoder and decoder. It must represent the entire source sequence in a fixed number of dimensions.
• The decoder uses the context vector as its initial state and generates tokens one at a time, conditioned on all previously generated tokens.
• Training uses cross-entropy loss with teacher forcing, where ground-truth previous tokens replace the decoder's own predictions to prevent error compounding during training.
• Bidirectional encoders process the source in both directions and consistently outperform unidirectional encoders by giving each position access to both left and right context.
• The fixed-size context vector creates a bottleneck that limits performance on long sequences, and the compression ratio worsens linearly as sequence length grows.
• BLEU score measures translation quality via n-gram precision with a brevity penalty. This provides an automatic and reproducible evaluation metric, though with known limitations for semantic evaluation.
• The framework generalizes beyond translation to summarization, code generation, image captioning, and any task that maps a variable-length source to a variable-length target.
• The seq2seq framework was the foundation for attention mechanisms and ultimately the transformer architecture, making it one of the most consequential architectural ideas in modern NLP.

The next chapter, Teacher Forcing, examines the training procedure in depth, including the exposure bias problem and scheduled sampling as a remedy. The Beam Search chapter then covers improved decoding strategies. Finally, Attention Intuition begins building up the mechanism that resolves the bottleneck problem at the heart of basic seq2seq models.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about the encoder-decoder framework.