IBM Statistical Machine Translation (1991) › Training from Data: The Birth of Corpus-Based NLP

The IBM approach changed natural language processing by learning entirely from data rather than human-encoded rules. The researchers assembled parallel corpora, collections of text in two languages where each sentence in one language had a corresponding translation in the other. These corpora came from diverse sources: the Canadian Hansard parliamentary proceedings, which by law had to be published in both English and French; technical manuals from multinational corporations that were translated into multiple languages; and news articles from international organizations.

The scale of data they used was remarkable for the time. The Canadian Hansard corpus alone contained millions of sentence pairs, providing the statistical foundation for learning reliable translation patterns. This represented a fundamental bet: that with enough examples, statistical patterns would emerge that captured the regularities of translation, even without understanding what those regularities were in linguistic terms.

The training process itself involved the expectation-maximization algorithm, a general technique from statistics for learning in the presence of hidden variables. In the E-step, the algorithm estimated the most likely alignments between words given the current translation probabilities. In the M-step, it updated the translation probabilities based on these estimated alignments. This cycle repeated until the model converged, typically after dozens of iterations through the entire training corpus.

To measure success, the researchers developed automatic evaluation metrics, including BLEU (Bilingual Evaluation Understudy). BLEU compared machine translations to one or more human reference translations, measuring how many sequences of words matched. While imperfect, BLEU provided an objective way to track progress without expensive human evaluation for every change to the system. This focus on quantitative evaluation became standard practice in machine translation and spread throughout natural language processing.