Phrase-based SMT & MERT (2003) › Minimum Error Rate Training: Optimizing for Evaluation

Statistical machine translation systems combine multiple features.phrase translation probabilities, language model scores, reordering costs, phrase penalties, and more.using log-linear models. Each feature has a weight that determines its importance, and these weights significantly impact translation quality. Early systems set these weights using maximum likelihood training, optimizing to maximize the probability of training data. However, maximum likelihood didn't necessarily produce the best translations according to evaluation metrics like BLEU. A system might achieve high likelihood while scoring poorly on BLEU, creating a mismatch between training objective and evaluation criterion.

Minimum Error Rate Training, introduced by Franz Josef Och in 2003, solved this problem by directly optimizing feature weights to maximize BLEU scores on development data. Instead of optimizing likelihood, MERT optimized the metric used for system evaluation. This alignment between training objective and evaluation metric improved results: systems tuned with MERT consistently achieved higher BLEU scores and better translation quality than those optimized for likelihood.

The MERT algorithm worked by iteratively improving feature weights. Given initial weights, the system generated translations for development sentences and computed BLEU scores. The algorithm then adjusted weights to increase BLEU, using a line search procedure along coordinate directions. For each feature, MERT found the weight value that maximized BLEU when all other weights were fixed, then moved to the next feature. This coordinate ascent procedure continued until convergence, producing weights that directly optimized translation quality as measured by BLEU.

The key insight was that MERT optimized weights for translation quality rather than an intermediate objective. Maximum likelihood training optimized the probability of seeing training translations, but this didn't guarantee good translations for novel inputs. MERT targeted the evaluator's measure: n-gram overlap with reference translations as a proxy for fluency and adequacy. By aligning optimization with evaluation, MERT enabled systems to achieve better practical performance.

MERT had important limitations that would motivate later research. The algorithm was computationally expensive, requiring many translation generations during optimization. It could be unstable, with small changes in development data leading to significantly different weights. Most critically, MERT scaled poorly to systems with many features, becoming impractical beyond about 20-30 features. This limited the complexity of feature sets that could be optimized effectively. Despite these limitations, MERT became standard practice for phrase-based systems and demonstrated the importance of optimizing for the right objective.

Alternative optimization methods would later address MERT's limitations. Margin-infused relaxed algorithms (MIRA) and variants like PRO used different optimization strategies that could handle more features and were more stable. These alternatives maintained MERT's core insight.optimizing for evaluation metrics.while improving scalability and stability. The fundamental principle established by MERT, that training objectives should align with evaluation metrics, became a guiding principle for machine translation and broader language AI research.