Phrase-based SMT & MERT (2003) › Transition to Neural Machine Translation

Phrase-based statistical machine translation dominated the field through the 2000s and into the early 2010s. Systems like Moses, built on phrase-based principles and tuned with MERT, achieved production-quality translations for many language pairs. Yet by the mid-2010s, neural machine translation began to demonstrate superior performance, eventually replacing statistical approaches for most applications.

Neural machine translation addressed many phrase-based limitations through learned representations and end-to-end learning. Instead of extracting discrete phrase pairs, neural systems learned continuous representations that could capture semantic relationships and long-range dependencies. Attention mechanisms, analogous to phrase alignment but learned automatically, allowed neural models to focus on relevant source words when generating each target word. The end-to-end training process optimized neural parameters to directly improve translation quality, similar to MERT's alignment between optimization and evaluation.

Remarkably, many neural translation concepts had statistical analogues. Attention mechanisms served functions similar to phrase alignment, identifying which source words were relevant for each target word. Neural language models, learned as part of the translation system, replaced separate n-gram language models. The encoder-decoder architecture, processing source sentences and generating targets, mirrored the noisy channel framework's separation of translation and language modeling.

Yet neural approaches also represented fundamental shifts. Learned representations replaced manual feature engineering. End-to-end optimization replaced the separate training of components like phrase tables and language models. Continuous vector spaces replaced discrete phrase and word vocabularies. These shifts enabled neural systems to capture patterns that phrase-based systems missed, particularly long-range dependencies, semantic relationships, and context-dependent translations.

The transition wasn't complete abandonment. Many neural systems incorporated phrase-based components, using phrase tables as additional features or initializing from phrase-based systems. The understanding developed during the phrase-based era.about alignment, reordering, multi-word units, and optimization.informed neural architecture design. MERT's principle of optimizing for evaluation metrics carried forward to neural training, where systems optimized metrics like BLEU directly rather than just likelihood.