IBM Statistical Machine Translation (1991) › The Statistical Translation Paradigm

The IBM approach rested on a deceptively simple insight: translation is fundamentally about finding correspondences between words and phrases across languages. Rather than attempting to understand meaning in some abstract sense and then re-express it in another language, the researchers focused on learning these correspondences directly from examples. If you could observe thousands or millions of sentences in two languages that meant the same thing, patterns would emerge showing how words and phrases systematically map from one language to another.

This led to a mathematical formulation borrowed from information theory and speech recognition. The researchers wanted to find the best translation by computing which target sentence was most likely given a source sentence. Using Bayes' theorem from probability theory, they expressed this as:

This equation, known as the noisy channel model, provided an elegant framework for translation. The intuition is counterintuitive at first: instead of directly modeling how to translate from source to target, the model asks "what target sentence might have been corrupted or transformed into this source sentence?" The source is treated as a "noisy" version of some original target sentence, and translation becomes the process of recovering the clean signal from the noise.

This formulation breaks the translation problem into two components. The term $P(\text{source}|\text{target})$ captures how target language sentences get transformed into source language sentences, learning word correspondences and structural changes. The term $P(\text{target})$ represents a language model that captures how likely a target sentence is in its own right, ensuring that translations are fluent and natural. By separating these concerns, the IBM researchers could learn each component from different data sources and combine them systematically.