BLEU Metric (2002) › The Evaluation Crisis in Machine Translation
History of Language AIAll chapters BLEU Metric (2002) › The Evaluation Crisis in Machine TranslationOne section · Sections in this chapter2 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
To understand why BLEU was so revolutionary, we need to appreciate the depth of the evaluation problem that existed before 2002. Machine translation research had been progressing since the 1950s, but by the early 2000s, the field faced a paradox: while researchers were developing increasingly sophisticated translation systems, they lacked a reliable way to measure whether their improvements made translations better.
Human evaluation, the traditional gold standard, required bilingual experts to read both the source text and the machine translation, then rate the translation's quality on various dimensions such as adequacy (does it preserve the meaning?) and fluency (does it read naturally?). This process was expensive and time-consuming. It was also fundamentally inconsistent. Two evaluators might disagree substantially on the quality of the same translation, particularly for subtle differences in style or when both translations were mediocre. Even the same evaluator might rate identical translations differently on different days, influenced by fatigue, context, or changing standards.
The inconsistency problem was compounded by the lack of standardization across research groups. Some teams used five-point scales, others used binary judgments. Some focused on adequacy, others on fluency. Some evaluated sentence by sentence, others looked at entire documents. This made it nearly impossible to compare results across different papers or reproduce previous findings. A system that scored well in one group's evaluation might perform poorly under another group's criteria, not because of any real difference in quality, but simply due to evaluation methodology.
Perhaps most critically, the slow pace of human evaluation created a bottleneck in the research process. Testing a single modification to a translation system might require weeks of human evaluation, making rapid experimentation impractical. Researchers needed a way to quickly test ideas, compare alternatives, and iterate on their systems. Without fast, reliable evaluation, the pace of progress in machine translation was artificially constrained by the evaluation process itself.