Encoder-Decoder Framework › End-to-End Training

The beauty of the encoder-decoder framework is that both components are trained jointly as a single model through standard backpropagation. There is no separate pre-training of the encoder or decoder; the entire model learns simultaneously to encode source sequences well and to decode those encodings into target sequences accurately.

This end-to-end training is one of the most important properties of the framework. In the earlier SMT era, the alignment model, the translation model, and the language model were trained separately on different objectives and then combined. Each component was optimized for a proxy objective that approximated the true goal (producing high-quality translations) but was not the actual goal. End-to-end training aligns every parameter in both the encoder and decoder directly with the translation objective, allowing the model to find internal representations that jointly optimize understanding and generation.

Training uses standard cross-entropy loss. Given a source sequence $x$ and a gold target sequence $y_1^*, y_2^*, \ldots, y_{T'}^*$, the model produces a probability distribution over the vocabulary at each target position. The loss is the negative log-likelihood of the correct tokens:

$$\mathcal{L} = -\sum_{t=1}^{T'} \log P(y_t^* \mid y_{< t}^*, x)$$

where:

• $T'$ is the length of the target sequence
• $y_t^*$ is the correct target token at position $t$
• $P(y_t^* \mid y_{< t}^*, x)$ is the model's probability assigned to the correct token

This is the standard cross-entropy loss averaged over positions, the same objective used for language model training. The model is trained to maximize the probability of the correct target sequence given the source.

One subtlety is that this loss decomposes as a product of conditional probabilities. The joint probability of the full target sequence is:

$$P(y_1, y_2, \ldots, y_{T'} \mid x) = \prod_{t=1}^{T'} P(y_t \mid y_{<t}, x)$$

Taking the log converts this product to a sum, which is the form we minimize. This factorization assumes that each target token, given the source and all previous correct target tokens, can be predicted as if it were an independent classification problem. This is exactly what teacher forcing enables during training.

During training, the decoder has two choices for what to use as the previous token input $y_{t-1}$. It could use its own prediction from the previous step, or it could use the ground-truth token. Using the model's own predictions is called autoregressive training; using the ground-truth tokens is called teacher forcing.

Teacher forcing is almost universally used during training because it dramatically accelerates convergence. If the model makes a mistake at position 3, that error would propagate through positions 4, 5, 6, etc. when using autoregressive training, causing a cascade of compounding errors. With teacher forcing, each position is trained independently on correct context, making the training signal much cleaner.

The downside is an exposure bias: the model never sees its own mistakes during training, so it can behave poorly when deployed, because it must now rely on its own (potentially incorrect) previous predictions. This training-inference mismatch is a known issue, and various remedies like scheduled sampling exist, where you gradually replace ground-truth tokens with model predictions as training progresses. The Teacher Forcing chapter covers these alternatives in depth.

In practice, training batches contain sequences of different lengths. To pack them into matrices for efficient GPU computation, shorter sequences are padded to the length of the longest sequence in the batch using a special <PAD> token. The loss is then masked so that padded positions do not contribute to the gradient. This ensures that padding does not confuse the model or artificially inflate the loss.

The masking step is important: without it, the model would be penalized for generating incorrect tokens at pad positions, which would distort the gradient and slow learning. In PyTorch, nn.CrossEntropyLoss accepts an ignore_index argument that automatically zeros out the loss at any position where the target label equals the specified index, making masking straightforward.

Pack-and-sort is another efficiency trick used in production training pipelines. Batches are sorted by source sequence length, and sequences within a batch are packed with minimal padding. The torch.nn.utils.rnn.pack_padded_sequence and pad_packed_sequence utilities support this workflow, allowing the LSTM to skip over padded positions entirely rather than processing them. This can save substantial compute for datasets with high variance in sequence length.

Training seq2seq models requires careful attention to optimization settings. A few considerations stand out.

First, gradient clipping is essential. Recurrent networks are prone to gradient explosions, where gradients grow exponentially as they propagate through many timesteps. Clipping the gradient norm to a maximum value (typically 1.0 or 5.0) prevents these instabilities. The Sutskever et al. paper used gradient clipping with a threshold of 5.0, and without it, training diverged.

Second, learning rate scheduling matters. Starting with a moderately high learning rate (around $1 \times 10^{-3}$ for Adam or $1.0$ for SGD with gradient clipping) and decaying it when validation loss stops improving is a common strategy. Transformer-based models introduced more sophisticated warmup schedules, but for RNN-based seq2seq, simple step decay or reduce-on-plateau typically suffices.

Third, initialization affects convergence. The Sutskever et al. 2014 model used uniform initialization in the range $[-0.08, 0.08]$ for all parameters and found this important for stable training. Modern frameworks use Xavier or He initialization by default, which is generally adequate.

Fourth, input sequence reversal is a somewhat surprising trick from the Sutskever et al. paper. They reversed the source sequence before feeding it into the encoder and observed substantial BLEU improvements. The intuition is that reversing brings the first source token closer to the first target token in the unrolled sequence of encoder-decoder steps, strengthening the gradient path between the two and making it easier for the decoder to learn to generate the beginning of the target sequence. The decoder must generate the target left to right, and with a reversed source, the last token the encoder sees is the first source token, which often corresponds to the first target token in translation. This shortens the path through the bottleneck for the tokens that matter most.