BLEU Metric (2002) › Why BLEU Succeeded

BLEU's adoption across the machine translation community was remarkably rapid, and this success stemmed from several key advantages that addressed the field's most pressing needs.

The most immediate benefit was speed. Computing a BLEU score required only simple string matching operations, which could be performed in milliseconds even on modest hardware. A researcher could evaluate thousands of translations in the time it would take a human evaluator to assess a single sentence. This dramatic acceleration in evaluation speed transformed the research workflow, enabling researchers to test variations of their systems multiple times per day rather than waiting weeks for evaluation results.

Consistency was equally important. BLEU always produced exactly the same score for the same input, eliminating the variability that plagued human evaluation. This made it possible to detect small improvements in translation quality with confidence. If a modification to a translation system increased the BLEU score by even a fraction of a point on a large test set, researchers could be certain this represented a real change rather than noise from inconsistent human judgments.

The metric's correlation with human judgments provided empirical validation. The original BLEU paper demonstrated that BLEU scores correlated strongly with human quality assessments when measured across different systems. While BLEU might disagree with human judges on individual sentences, the ranking of systems by BLEU score tended to match the ranking by human preference. This meant researchers could use BLEU for rapid iteration while still trusting that improvements in BLEU score would likely correspond to improvements that humans would appreciate.

BLEU's language independence made it universally applicable. The same algorithm worked for translating between any pair of languages, from English to French, Chinese to Arabic, or Japanese to German. Researchers didn't need to develop language-specific evaluation metrics or hire evaluators for each language pair. As long as they had reference translations in the target language, BLEU could assess translation quality.

Perhaps most importantly, BLEU provided a common standard that made research comparable across groups and reproducible over time. When different papers reported BLEU scores on the same test sets, readers could directly compare the systems' performance. This standardization accelerated progress by making it clear which approaches were most promising and enabling researchers to build on each other's work with confidence.