IBM Statistical Machine Translation (1991) › The IBM Models: A Progression of Sophistication

The IBM researchers didn't develop a single translation system, but rather a sequence of five models, each building on its predecessors to capture increasingly subtle aspects of how languages correspond. This progression reflected a deliberate strategy: start with the simplest possible assumptions, then systematically add complexity to address observed limitations.

IBM Model 1 established the foundation by making the strongest simplifying assumptions. It treated translation as purely a matter of word-to-word correspondences, ignoring word order entirely. While this might seem overly naive, it provided a tractable starting point for learning which words in one language tend to align with which words in another. More importantly, it introduced the mathematical machinery that all subsequent models would build upon.

IBM Model 2 took the first step toward realism by acknowledging that word position matters. Languages differ not just in vocabulary but in how they order words. In English we say "the red car," while in French it's "la voiture rouge" with the adjective following the noun. Model 2 introduced alignment probabilities that captured tendencies about which positions in the source sentence typically align with which positions in the target sentence.

IBM Model 3 addressed another fundamental limitation: the assumption that each source word produces exactly one target word. In reality, a single word might translate to multiple words, or disappear entirely. The English word "children" might translate to Spanish "los niños," two words where English had one. Model 3 introduced fertility parameters that captured how many target words each source word typically generates.

IBM Model 4 refined how word positions change during translation through more sophisticated distortion modeling. Rather than treating position changes as independent, Model 4 recognized that related words tend to move together. If a noun moves to a different position, its modifying adjectives typically move with it in predictable ways.

IBM Model 5 made the distortion modeling even more refined and improved the numerical stability of the training process. By this point, the model captured a remarkably rich set of translation phenomena while remaining computationally tractable.

Each model in this sequence represented a tradeoff between realism and computational complexity. The researchers could have started with the most sophisticated model, but the progression served a practical purpose: simpler models could be trained more reliably, and their parameters could initialize the more complex models, making the entire training process more stable and effective.