BLEU Metric (2002) › The Limitations Become Apparent

As BLEU gained widespread adoption, researchers also became increasingly aware of its limitations. These shortcomings didn't diminish BLEU's practical value for rapid system comparison, but they highlighted important gaps between automatic metrics and true translation quality.

The most fundamental limitation stemmed from BLEU's focus on surface-level n-gram matching. BLEU treats words as atomic symbols without understanding their meaning, so it cannot recognize synonyms or semantically equivalent phrases. A translation using "big" instead of "large" would be penalized even though the meaning is nearly identical. More problematically, BLEU cannot distinguish between translations that preserve meaning and those that subtly distort it. A translation might use exactly the right words in almost the right order, achieving a high BLEU score, while completely reversing the meaning of a negation or misidentifying the subject and object of an action.

The quality of BLEU scores depends critically on the reference translations. With only a single reference, BLEU severely penalizes any legitimate variation in wording or phrasing. Multiple references help, but even with several references, BLEU cannot account for all valid translation possibilities. Two human translators working independently might produce quite different translations that are both excellent, yet a machine translation matching one would score poorly when compared against the other. This reference dependence means that BLEU measures not absolute translation quality but rather similarity to a specific set of reference translations.

BLEU exhibits various biases that can advantage certain types of systems over others. The brevity penalty addresses length bias in one direction (preventing too-short translations) but does nothing to penalize translations that are too long and wordy. The equal weighting of different n-gram lengths means that systems can potentially game the metric by optimizing for specific n-gram lengths. More subtly, BLEU tends to favor translations that are structurally similar to the references, potentially penalizing valid translations that use different but equally correct grammatical structures.

The metric also struggles with interpretability. Unlike human evaluation with clear quality scales (like "excellent" or "poor"), BLEU produces scores between 0 and 1 that lack intuitive meaning. Is a BLEU score of 0.35 acceptable? What about 0.42? The answer depends on the language pair, domain, difficulty of the source text, and number of references. Two systems with BLEU scores of 0.38 and 0.40 might produce noticeably different translation quality, or the difference might be imperceptible. Without extensive experience with BLEU scores in a particular setting, the numbers themselves provide limited insight.

Perhaps most importantly, BLEU's limitations mean it should not be the sole metric for evaluating translation quality. While invaluable for rapid iteration during development, final evaluation of systems should include human assessment, particularly for high-stakes applications where translation errors could have serious consequences. BLEU guides researchers toward better systems but cannot replace human judgment about whether a translation is truly fit for purpose.