Encoder-Decoder Framework › Practical Considerations for Real Translation

Our implementation above is simplified to aid learning. Real production-scale seq2seq systems for machine translation involve several additional components and considerations.

A toy model trained on 20 sentences illustrates the mechanics but cannot generalize. Real translation models require millions to hundreds of millions of parallel sentence pairs. The WMT (Workshop on Machine Translation) datasets for English-French, English-German, and English-Czech are standard benchmarks, containing 40 million, 4.5 million, and 1 million pairs respectively. Collecting and cleaning such datasets is a major engineering effort: web-scraped parallel data from multilingual websites, Europarl transcripts, UN documents, and translated news articles are all used.

Data quality matters as much as quantity. Noisy pairs, where the source and target are not translations of each other, degrade model quality. Deduplication, length-ratio filtering, and language identification are standard preprocessing steps that remove the most egregiously misaligned pairs.

Word-level vocabularies create the out-of-vocabulary problem: words that do not appear in training are mapped to <UNK>, losing their meaning entirely. Character-level models as in our implementation avoid this but must model very long sequences (a single word may span multiple timesteps), which strains the context vector.

Byte-pair encoding (BPE) and its variants solve this by learning a subword vocabulary of 30,000 to 64,000 pieces, where common words appear as single tokens and rare words are split into common subword pieces. This allows the model to handle any word in any language by composing subword pieces, while keeping sequence lengths manageable. We covered BPE in depth in the Subword Tokenization part of this book. For translation, both the source and target vocabularies may be trained jointly on the combined parallel corpus, creating a shared subword vocabulary that facilitates transfer between the two languages.

Greedy decoding is convenient but leaves quality on the table. Beam search with a beam width of 4 to 10 typically improves BLEU by 2 to 5 points compared to greedy decoding on standard benchmarks. The computational cost scales linearly with beam width, since you are maintaining $k$ parallel decoder states instead of one. For real-time translation systems, beam width is a tunable latency-quality tradeoff: wider beams are better but slower.

Beam search also introduces a length normalization issue. Because beam search multiplies probabilities across timesteps, longer sequences accumulate more probability terms and tend to have lower total probability than shorter sequences, causing the model to prefer unnaturally short translations. Dividing the sequence log-probability by the sequence length (possibly raised to a power less than 1 to soften the normalization) counteracts this length bias.

Once you have a working seq2seq framework, extending it to multiple language pairs is relatively straightforward. The Google Multilingual Neural Machine Translation (MNMT) system demonstrated that a single seq2seq model could translate between dozens of language pairs simultaneously, simply by prepending a target language tag to the source sequence (for example, <2fr> to indicate "translate to French"). This massively multilingual approach allows the model to transfer representations across related languages, improving translation quality for low-resource language pairs that benefit from cross-lingual sharing.