BLEU Score › Limitations and Impact

BLEU revolutionized machine translation by providing a standardized, automatic evaluation metric that enabled rapid iteration. Before BLEU, researchers relied on expensive and slow human evaluations, limiting experimental throughput. BLEU allowed nightly regression testing and systematic optimization, directly contributing to the rapid improvement of statistical and later neural machine translation systems. The metric facilitated the development of shared tasks and benchmarks, creating a competitive research environment that accelerated progress from 2002 through the deep learning revolution.

However, BLEU suffers from significant limitations that have become more apparent as the field evolved. Understanding these limitations helps you avoid misuse of BLEU and understand what automatic evaluation can and cannot tell you about a generation system.

The most glaring limitation is that BLEU is semantically blind. BLEU treats words as atomic tokens with no understanding of meaning. As we saw in our example, "cat" and "feline" produce no match despite being synonyms. Similarly, "quickly" and "fast" are treated as completely different words. This penalizes valid translation variation and fails to capture meaning preservation when word choice differs. The metric cannot distinguish between a translation that preserves meaning using different words and one that completely mistranslates the content, provided the surface forms do not match the reference.

This semantic blindness becomes more damaging for languages with rich morphology. In German, the same noun can take many different inflected forms depending on case and number. A translator who correctly identifies the right grammatical form but differs from the reference inflection receives no credit, even though the meaning is preserved. In agglutinative languages like Finnish or Turkish, the situation is even more extreme: a single word in these languages can correspond to an entire phrase in English, and matching that word exactly may be impossible even when the translation is semantically perfect.

The limitations are not merely theoretical. Studies have repeatedly shown cases where BLEU strongly disagrees with human judgment because it misses valid paraphrases. A translation system that learned to always echo the exact phrasing of its training references would score higher on BLEU than one that produced more natural, varied output, even though human evaluators would prefer the latter.

BLEU also struggles with word order flexibility. While higher-order n-grams nominally check fluency, they penalize perfectly valid reorderings. In many languages, word order differs substantially from English; a translation might be fully fluent and accurate yet receive a low BLEU score because phrases appear in different positions than in the reference. German, for instance, often places verbs at the end of clauses, while Japanese follows a subject-object-verb order that requires extensive reordering when translating to English. BLEU unfairly penalizes translations that respect target language syntax when it differs from the specific reference translation provided.

Consider translating from a free-word-order language like Russian or Latin. The same sentence can appear with the subject before or after the verb, with adjectives in various positions relative to their nouns, and with prepositional phrases in multiple valid locations. A human translator would choose the most natural English order based on information structure and focus, which might differ from what the reference translator chose. BLEU would penalize both as equivalent errors even though one might be significantly more natural.

The metric exhibits weak correlation with human judgment at the sentence level. While corpus-level BLEU roughly tracks system rankings, individual sentences can have terrible BLEU scores despite being good translations, or high BLEU scores while being ungrammatical. This makes BLEU unsuitable for applications requiring reliable sentence-level quality estimation, such as filtering training data or confidence estimation. A translation might be perfectly adequate but receive zero BLEU because it uses a completely valid paraphrase not present in the limited reference set.

The correlation problem is compounded by reference sparsity. Standard test sets typically include one to four reference translations per source sentence. Even with four references, a professional translator would still produce variations not captured in the reference set. The more creative or varied the correct translation space, the worse BLEU's correlation with human judgment becomes. For poetry, idiomatic expressions, or culturally embedded text, the space of valid translations can be enormous, and the handful of references in a test set cannot adequately represent it.

Domain sensitivity presents another challenge. BLEU scores are not comparable across different language pairs or domains. A BLEU of 30 might represent state-of-the-art for Chinese-English translation but poor quality for Spanish-English, where linguistic similarities allow for higher scores. Even within a language pair, scores vary by genre: news translation typically achieves higher BLEU than technical manuals or literary text, because news language is more formulaic and predictable, leading to more reference overlap.

This has led to the unfortunate practice of "BLEU score inflation" where papers report seemingly high numbers that are unimpressive when accounting for dataset difficulty. You must normalize scores against baseline systems to make meaningful comparisons. A new neural architecture claiming a BLEU of 45 on a news translation task is far less impressive if the previous best system already achieved 44, whereas the same absolute number on a difficult literary translation task might represent a substantial breakthrough.

Perhaps the deepest limitation of BLEU is what happens when it becomes the primary optimization target. Goodhart's Law states that when a measure becomes a target, it ceases to be a good measure. Research groups began optimizing specifically for BLEU: developing smoothing methods tuned to maximize BLEU while producing less fluent text, selecting training data that maximized reference overlap, and designing architectures that were good at n-gram matching rather than semantic preservation. Some researchers noted that BLEU-optimized systems could produce translations that scored well on the metric but were judged as significantly worse by human evaluators who assessed overall meaning and naturalness.

The phenomenon of "BLEU hacking" manifested in several ways. Minimum Bayes risk (MBR) decoding, which selects the translation most similar to a set of candidates, was shown to produce outputs that scored well on BLEU but were bland and repetitive. Certain beam search strategies produced translations that successfully matched common n-grams by repeating safe phrases. The research community gradually recognized that maximizing BLEU was not the same as maximizing translation quality, leading to calls for supplementary metrics and more rigorous human evaluation protocols.