BLEU Score › BLEU Score

How do you measure the quality of a machine translation when there are many correct ways to translate the same sentence? Unlike classification tasks with a single right answer, text generation produces open-ended outputs where multiple phrasings can be equally valid. This fundamental challenge plagued early research in statistical machine translation during the 1990s. Researchers found themselves trapped in a difficult methodological loop: to improve translation systems, they needed to evaluate them, but human evaluation required fluent bilingual speakers, cost significant money, and took days or weeks to complete. This bottleneck severely limited experimental iteration, as researchers could only test a handful of configurations per study.

The Bilingual Evaluation Understudy (BLEU) score, introduced by Papineni et al. in 2002, emerged as the solution that dominated the field for two decades. BLEU approaches evaluation through a deceptively simple intuition: good translations share words and phrases with professional human translations. By counting matching n-grams between a candidate translation and one or more reference translations, BLEU provides a numerical score that correlates reasonably well with human judgment while remaining computationally cheap. The brilliance of BLEU lies in its recognition that while we cannot easily quantify "good translation" abstractly, we can operationalize the concept through surface-level lexical overlap with high-quality reference translations.

BLEU combines two core components: modified n-gram precision to measure accuracy, and a brevity penalty to discourage overly short translations. As we discussed in Part II, Chapter 2: N-grams, n-grams capture local word order and collocation patterns. BLEU extends this idea by comparing n-grams between machine output and human references, checking unigram overlap along with bigrams, trigrams, and 4-grams to ensure fluency and grammatical coherence. This multi-scale approach acknowledges that translation quality operates at multiple levels: individual word choice matters, but so does the proper pairing of adjacent words and the maintenance of longer phrase structures.