Google Neural Machine Translation (2016) › The Problem

The traditional approach to machine translation, which had dominated the field since the 1990s, relied on statistical phrase-based methods that broke sentences into phrases, translated each phrase independently, and then reordered the translated phrases to form coherent sentences. This approach, while effective for its time, had fundamental limitations that became increasingly apparent as researchers pushed for better translation quality.

The phrase-based models often produced translations that were grammatically correct but lacked fluency and naturalness. A sentence might be perfectly intelligible and contain all the necessary information, yet still sound unnatural to native speakers. The systems struggled with long-range dependencies and context, as they processed phrases in isolation without considering the broader context of the sentence or document. Words that referred back to earlier parts of the sentence, or whose meaning depended on subsequent context, were often translated incorrectly.

Consider a sentence like "The company announced that it would increase prices." A phrase-based system might translate "it" without recognizing that it refers to "the company," leading to awkward or incorrect translations. The system processed "it would increase prices" as an isolated phrase, losing the connection to the subject established earlier in the sentence. This limitation was particularly problematic for languages with different word orders or grammatical structures that required careful attention to sentence-level context.

The phrase-based approach also required extensive feature engineering and manual tuning. Researchers had to design features that captured linguistic patterns, translation probabilities, and reordering rules. Each language pair required careful calibration, making it difficult and time-consuming to adapt systems to new language pairs or domains. The complexity of feature engineering meant that adding support for a new language pair was a significant undertaking, requiring both linguistic expertise and engineering effort.

Additionally, statistical systems struggled with rare words and out-of-vocabulary terms. Words that appeared infrequently in the training data were often handled poorly, and completely unseen words could cause translation failures. The systems relied on surface-level patterns and phrase alignments, which worked well for common phrases but failed when encountering novel constructions or domain-specific terminology.