Encoder-Decoder Framework › Seq2seq Applications

The encoder-decoder framework generalizes well beyond machine translation. Any task that maps one variable-length sequence to another is a natural fit.

Translation was the motivating application and remains the canonical benchmark. Given a source sentence in one language, the decoder generates a sentence in the target language. The model must implicitly learn the vocabulary, grammar, and semantics of both languages from parallel training corpora. What makes this particularly remarkable is that the model has no explicit alignment table or phrase dictionary: it learns to align source and target concepts implicitly through the continuous representations in the context vector and, once attention is added, through the attention weights.

One practical challenge is vocabulary coverage. Real translation requires handling rare words, proper nouns, technical terms, and code-switching between languages. Early seq2seq models used fixed vocabularies of around 30,000 to 80,000 tokens, and words outside this vocabulary were mapped to a single <UNK> (unknown) token. This severely limited quality on any sentence with rare terminology. The development of subword tokenization, and later byte-pair encoding (as covered in the Subword Tokenization part of this book), addressed the vocabulary problem by breaking rare words into frequent subword units that the model can handle.

In summarization, the source is a document and the target is a shorter summary. Abstractive summarization uses an encoder-decoder model where the decoder generates new summary text rather than extracting source sentences verbatim. The source is typically much longer than the target, pushing the context vector bottleneck to its limits, which is why attention-based models dramatically outperform basic seq2seq on this task.

An additional challenge in summarization is that the source document may contain specific entities, numbers, and named facts that must appear verbatim in the summary. The copy mechanism, which allows the decoder to directly copy tokens from the source rather than generating them from the vocabulary, was developed specifically to address this need. The copy mechanism can be viewed as a special attention pattern that, instead of building a weighted average of encoder hidden states, selects a source position directly and outputs the token at that position.

Code generation maps a natural language description (the source) to code (the target). This is another natural seq2seq task, though the output vocabulary and structure differ significantly from natural language: code has strict syntactic rules and the decoder must track things like indentation, brackets, and variable names across long spans.

The deterministic structure of programming languages creates both opportunities and challenges. On one hand, it is easier to evaluate correctness objectively (does the code run and produce the right output?) than to evaluate translation quality. On the other hand, a single misplaced bracket or wrong indentation level produces code that fails entirely, making precision more important than in natural language generation where small errors are often tolerable.

Seq2seq models have been applied to question answering (question as source, answer as target), dialogue systems (user utterance as source, response as target), speech recognition (acoustic features as source, text as target), and image captioning (image features as source, caption as target). The framework is remarkably versatile.

For image captioning, the encoder is replaced with a convolutional neural network that produces a fixed-size feature vector from the image, and this feature vector plays the role of the context vector that initializes the decoder. The decoder then generates a natural language caption. The "source" in this case is not a sequence at all, but the encoder-decoder interface remains the same: compress the input into a fixed representation, then decode into a sequence. This flexibility shows that the framework's core idea, separating representation from generation, is not specific to sequence inputs.