IBM Statistical Machine Translation (1991) › Challenges and Limitations: The Boundaries of Statistics

For all their success, the IBM models revealed fundamental limitations that would drive the next generation of research. These weren't mere implementation details that could be fixed with better engineering, but rather reflected the core assumptions underlying the statistical approach.

The most significant limitation was the focus on word-level modeling. By treating translation primarily as a matter of word-to-word correspondences, the models missed larger patterns at the phrase and sentence level. Idiomatic expressions like "kick the bucket" don't translate word by word; they need to be recognized and translated as units. The models had no natural way to capture that "break" and "leg" in "break a leg" shouldn't be translated literally when wishing someone luck.

The models also made largely local decisions. When translating a word, they considered primarily nearby words and their alignments, missing the global coherence of the sentence. A translation might get each word or phrase roughly right in isolation, yet produce an awkward or even incomprehensible sentence overall. Human translators maintain awareness of the entire sentence and document context; the IBM models had no mechanism for this kind of global coordination.

Data sparsity posed another challenge. Even with millions of training sentences, most possible word combinations appeared rarely or not at all. If "purple" and "elephant" never appeared together in the training data, the model had no direct evidence about how to translate "purple elephant." While the model could fall back on its separate knowledge of "purple" and "elephant," this led to less reliable translations for rare or novel combinations.

The limited context window was also problematic. The models could consider only a small window of surrounding words when making translation decisions. But translation often requires understanding relationships that span entire sentences or even across sentences. Pronouns need to agree with nouns that might appear many words earlier, and ambiguous words need to be disambiguated based on broader context.