BLEU Metric (2002) › The Evolution of Evaluation Metrics

BLEU's limitations motivated researchers to develop more sophisticated evaluation metrics that could address its weaknesses while maintaining its practical advantages. Each subsequent metric attempted to capture aspects of quality that BLEU missed, creating a more detailed picture of generated-text quality.

METEOR (Metric for Evaluation of Translation with Explicit ORdering), introduced in 2004, made several improvements to BLEU's approach. Most importantly, METEOR included synonym matching through WordNet, allowing it to recognize that "big" and "large" convey similar meanings. It also incorporated stemming to match different forms of the same word ("running" and "runs") and considered word order through alignment penalties. By balancing precision and recall rather than focusing solely on precision, METEOR provided a more complete picture of translation quality. These enhancements came at the cost of some additional complexity and language-specific resources, but METEOR often correlated better with human judgments than BLEU.

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) adapted BLEU's principles specifically for summarization tasks. While BLEU focuses on precision (what fraction of the generated n-grams appear in references), ROUGE emphasizes recall (what fraction of the reference n-grams appear in the generated text). This makes sense for summarization, where the goal is to ensure the summary captures key information from the references rather than to avoid extraneous content. ROUGE became the standard metric for summarization research, demonstrating that task-specific adaptation of evaluation principles often works better than applying a general-purpose metric.

The rise of neural language models enabled a qualitative leap in evaluation methodology. BERTScore, introduced in 2019, uses contextual embeddings from BERT to measure semantic similarity between generated and reference text. Instead of requiring exact n-gram matches, BERTScore computes similarity between contextualized word representations, allowing it to recognize paraphrases and semantic equivalence that BLEU would miss. This neural approach to evaluation aligned much better with human judgments than traditional n-gram metrics, though it required significantly more computation.

BLEURT took this further by fine-tuning BERT specifically for the evaluation task, training it to predict human quality ratings. By learning from large datasets of human judgments, BLEURT could capture subtle aspects of quality that hand-designed metrics miss. This represented a shift from rule-based evaluation to learned evaluation, where the metric itself is a neural model trained to approximate human judgment.

Despite these advances, human evaluation remains the ultimate standard. Automatic metrics serve as proxies for human judgment, useful for rapid development but not substitutes for careful human assessment. The most rigorous evaluations combine multiple automatic metrics with structured human evaluation, recognizing that each perspective captures different aspects of quality.