Phrase-based SMT & MERT (2003) › The Limitations of Word-Based Translation

The IBM statistical translation models of the early 1990s revolutionized machine translation by demonstrating that translation patterns could be learned from parallel data. The IBM models operated at the word level, learning translation probabilities for individual words and using these probabilities, combined with alignment models, to generate translations. This approach achieved unprecedented success compared to rule-based systems, but it faced fundamental limitations that became increasingly apparent as researchers pushed for higher translation quality.

The most significant limitation was the word-level granularity itself. Natural language contains many multi-word units that don't translate word by word. Idiomatic expressions are the clearest examples: "kick the bucket" means "to die," but translating each word individually would produce nonsense. Even non-idiomatic phrases often require multi-word translation units. The French phrase "je ne sais pas" translates to "I don't know," but this correspondence operates at the phrase level, not word by word. Word-based systems struggled with such patterns, either missing them entirely or producing awkward literal translations.

Collocations and fixed expressions also caused problems. Common multi-word combinations like "hot dog," "machine learning," or "United Nations" have specific translations that can't be derived from individual word translations. Word-based systems might translate "hot" and "dog" separately, missing that "hot dog" is a single lexical unit with a specific meaning. Similarly, compound nouns in languages like German require phrase-level treatment. The word "Maschinenlernen" (machine learning) needs to be treated as a unit when translating to English, not decomposed into its component parts.

Word order issues compounded the problem. The IBM models included distortion or reordering models to handle word order differences between languages, but reordering at the word level was often unnatural. Consider translating "the red car" from English to French, where it becomes "la voiture rouge." Word-based systems might translate "the" → "la," "red" → "rouge," "car" → "voiture," then attempt to reorder. But phrase-based systems could learn that "the red car" as a unit translates to "la voiture rouge," capturing both the translation and the word order in a single phrase pair. This was more natural and often more accurate.

Data sparsity problems became more severe at the word level. Even with large parallel corpora, many word sequences appeared rarely or never. A word-based system might have good translation probabilities for individual words but poor estimates for their combinations. If "purple elephant" never appeared in training data, the system would translate it based on separate knowledge of "purple" and "elephant," potentially missing language-specific collocations or conventions. Phrase-based systems could learn that certain word combinations form coherent translation units even when the exact combination is rare, by learning from similar phrases and patterns.

The reordering problem was also more complex at the word level. Languages differ significantly in how they order words, and word-level reordering models needed to capture many specific patterns. A phrase-based system could learn reordering at the phrase level, which was often simpler and more linguistically natural. For example, a phrase-based system might learn that English adjective-noun phrases typically become noun-adjective in French, capturing this pattern more directly than word-level reordering models.