Seq2Seq for MT (2014) › The Problem: Translation as a Jigsaw Puzzle With Missing Pieces

Imagine you're trying to translate "The cat sat on the mat" into French. Seems straightforward, right? But consider what a statistical machine translation system from 2013 had to do to accomplish this seemingly simple task.

First, it would break the English sentence into chunks called phrases: "the cat", "sat on", "the mat". Then it would look up each phrase in a massive phrase table.essentially a dictionary storing millions of translation pairs extracted from parallel text. Maybe "the cat" had been seen before and could translate to "le chat". Maybe "sat on" became "assis sur". So far, so good.

French word order complicated the process. The phrase-based system needed explicit reordering rules to shuffle the translated phrases into grammatically correct French. These rules were hand-crafted by linguists who understood how English and French sentence structures differed. Want to translate English to Japanese instead? You'd need entirely different reordering rules, because Japanese puts verbs at the end of sentences. Every language pair required its own set of carefully engineered transformation rules.

And that was just the beginning. The system also needed a language model to ensure the output sounded fluent, an alignment model to track which source words corresponded to which target words, and a scoring function to choose between multiple possible translations. Each component was trained separately, optimized independently, then somehow integrated into a coherent pipeline. It worked, but it was fragile, expensive to build, and required deep linguistic expertise to extend to new language pairs.

Statistical machine translation wasn't really a single system.it was more like a factory assembly line with a dozen different stations, each performing one specialized task:

• Word alignment: Figure out which English words correspond to which French words in the training data
• Phrase extraction: Identify common multi-word sequences and their translations
• Translation model: Calculate probabilities for phrase translations
• Reordering model: Determine how to shuffle phrases to match target language grammar
• Language model: Make sure the output sounds fluent in the target language

Each component was trained separately, optimized independently, and often used completely different statistical techniques. The alignment model might use one approach, while the language model used another. Getting these pieces to work together coherently required extensive manual tuning.adjusting weights and parameters across all the components, hoping that improving one piece wouldn't break another.

This modular architecture created a fundamental problem: there was no way to optimize the entire system end-to-end. When the final translation came out wrong, which component should you fix? Maybe the phrase table had the wrong translation. Or maybe the phrase was correct but the reordering was wrong. Or maybe both were fine but the language model preferred a worse-sounding output. The system couldn't learn from its mistakes in a unified way. It was like trying to improve a relay race by training each runner separately, never letting them practice passing the baton.

Consider translating the English word "bank" into Spanish. Should it become "banco" (financial institution) or "orilla" (river bank)? The answer depends on context from elsewhere in the sentence.maybe even from several words away. But phrase-based systems made translation decisions locally, looking only at nearby words within each phrase. They couldn't see the bigger picture.

This limitation became particularly problematic for phenomena like pronoun agreement. In the sentence "The developer finished her code and pushed it to the repository," you need to understand that "her" refers to "developer" and "it" refers to "code" to translate correctly into languages with gendered nouns. But if these words ended up in different phrases, the system might lose this long-distance connection. The translation would be grammatically wrong, and there was no easy way to fix it without redesigning the entire pipeline.

By 2014, competitive statistical translation systems incorporated dozens of hand-crafted features:

• How long is the source sentence versus the target?
• How many times does this phrase appear in the training data?
• What's the alignment score between source and target phrases?
• Does this translation preserve named entities correctly?
• How well does the output match n-gram patterns in target language text?

Each feature represented some linguistic insight about what makes good translations. But here's the catch: these features worked for some language pairs and not others. Features that helped English-French translation might hurt English-Chinese. Building a new translation system meant starting from scratch with feature engineering, testing countless combinations to find what worked.

The complexity grew organically over time. A researcher would notice a systematic error.maybe the system consistently mistranslated passive voice.and add a feature to fix it. Another researcher would spot a different problem and add another feature. The system became a patchwork of fixes, each addressing a specific issue but none addressing the underlying problem: the system couldn't learn what mattered on its own. It was like trying to teach someone to play piano by giving them a thousand individual rules about which keys to press, rather than letting them develop an intuitive understanding of music.

What happens when you need to translate a word the system has never seen? In 2014, if a phrase-based system encountered "cryptocurrency" but had never seen that word during training, it would typically either copy it literally into the output or replace it with a generic "unknown word" marker. Neither option was ideal.

For proper nouns like "Michael" or "Tokyo", copying might work if both languages used similar names. But for technical terms, neologisms, or words specific to particular domains, this behavior created serious problems. A medical translation system trained on general news text would fail spectacularly when asked to translate a medical research paper full of specialized terminology it had never encountered. The system had no way to reason about word structure or meaning.it could only look up what it had memorized.