BLEU Score › Historical Context and Motivation

To appreciate why BLEU mattered so much when it appeared, you need to understand the state of machine translation evaluation in the late 1990s. Statistical machine translation (SMT) systems were rapidly improving, but the research community lacked a shared yardstick for measuring progress. Different research groups used different evaluation protocols, sometimes reporting results on proprietary test sets, making it nearly impossible to compare systems across institutions or determine which architectural choice drove improvements.

Human evaluation, when it did occur, was carried out in varying formats: adequacy ratings on a 5-point scale, fluency ratings on a separate scale, or preference judgments where annotators chose between two system outputs. These formats did not always agree with each other, and even the same format produced highly variable results depending on annotator selection, briefing procedures, and fatigue effects. A five-point adequacy scale used in one laboratory might bear little resemblance to what another laboratory considered a "4 out of 5" translation. Interlabeler agreement was poor, reproducibility was low, and the community had no way to determine whether a new method was better or simply happened to be evaluated more favorably.

Papineni and colleagues were working at IBM Research, at that time one of the centers of statistical machine translation research. Their goal was pragmatic: they needed to run dozens of experiments per day, comparing variants of word alignment models, language models, and decoding algorithms. Human evaluation at that scale was economically impossible. They set out to find an automatic metric that would correlate with human judgment well enough to serve as a reliable proxy during development, even if it was not perfect.

The key insight in the BLEU paper was methodological as much as technical. Rather than trying to build a perfect evaluation metric, Papineni et al. asked a more tractable question: can we build a metric that, when comparing two systems on the same test set, almost always agrees with human evaluators about which system is better? System-level ranking turns out to be far easier than absolute quality assessment. This design decision, targeting ranking agreement rather than absolute correlation, explains why BLEU works reasonably well for its intended purpose while failing badly in many situations where absolute quality matters.

The paper demonstrated that BLEU correlated highly with human judgments across multiple language pairs and test sets, reporting rank correlation coefficients above 0.99 for system-level comparisons. This remarkable number convinced the community to adopt the metric, and within a few years, reporting BLEU scores became essentially mandatory for machine translation publications. Shared evaluation tasks like NIST and later WMT (Workshop on Machine Translation) standardized on BLEU, creating the competitive benchmarking infrastructure that drove rapid progress through the mid-2000s.