ROUGE & METEOR (2004) › The Problem: BLEU's Limitations in New Domains

BLEU's success in machine translation evaluation had established it as the dominant automatic evaluation metric by the early 2000s. Its advantages were clear: it was fast, reproducible, and correlated reasonably well with human judgments of translation quality. However, as researchers applied BLEU to new tasks and domains, its limitations became increasingly apparent. These limitations weren't necessarily flaws in BLEU's design.they reflected fundamental assumptions that worked well for translation but broke down in other contexts.

The most significant limitation for summarization tasks was BLEU's precision-oriented focus. BLEU measured what fraction of the generated text's n-grams appeared in reference translations. This emphasized that good translations shouldn't introduce extraneous content. This made sense for translation, where adding words not present in the source could indicate hallucination or errors. But summarization had different goals: the objective was to capture all important information from a source document, even if that meant generating content not present in any single reference summary. Different human summarizers might emphasize different aspects of the source, producing reference summaries that varied substantially in their specific wording. A good automatic summary needed to capture the main information, not match a particular reference's exact wording.

This created a recall problem. If a generated summary captured important information that happened to be expressed differently than in the reference, BLEU would give it low scores even if the summary was high quality. A summary that perfectly captured the main points but used different phrasing would score poorly. This was fundamentally backwards for summarization, where the goal was information coverage rather than precise wording. Summarization needed a metric that emphasized whether the summary captured important reference content, not whether it precisely matched reference wording.

Another fundamental limitation was BLEU's inability to recognize semantic equivalence. BLEU required exact n-gram matches, meaning it couldn't recognize that "big" and "large" conveyed the same meaning, or that "automobile" and "car" referred to the same concept. This semantic blindness meant that two semantically equivalent translations could receive very different BLEU scores depending on their specific word choices. A translation that used synonyms might score lower than one using exact reference words, even if both conveyed the same meaning.

The morphological variation problem presented another challenge. BLEU treated "running," "runs," and "ran" as completely different words, failing to recognize their shared root. This meant translations using different grammatical forms could receive different scores even when expressing equivalent meanings. Languages with rich morphological systems, where a single root might appear in dozens of forms, faced particular challenges under BLEU's strict matching.

Word order sensitivity was another limitation. BLEU considered word order through n-gram matching, but it had no explicit mechanism for penalizing severely scrambled word order beyond the fact that scrambled orders would have fewer matching n-grams. A translation that got all the right words but in completely wrong order might still receive some n-gram credit, particularly for unigrams. BLEU lacked a principled way to balance the importance of word choice versus word order.

These limitations weren't just theoretical concerns. They manifested in real-world evaluation failures where systems with good semantic understanding scored poorly, or where different valid paraphrases received wildly different scores. They created incentives for systems to optimize for exact word matching rather than true semantic quality, potentially leading researchers to develop systems that looked good on metrics but failed in actual use. The field needed evaluation metrics that could recognize quality beyond exact string matching.