BLEU Metric (2002) › Principles That Endured

Looking beyond BLEU's specific formula, the metric established several principles that would shape evaluation methodology throughout natural language processing. These principles proved more durable than the metric itself, influencing how researchers think about evaluation even in contexts where BLEU is inappropriate.

The principle of automatic evaluation fundamentally changed research practice. BLEU demonstrated that carefully designed automatic metrics could provide reliable signal about system quality, even if imperfect. This encouraged researchers to invest in developing automatic metrics for other tasks, accelerating progress across the field. The cycle of development became: use automatic metrics for rapid iteration, validate with human evaluation periodically, and refine automatic metrics to better align with human judgments. This workflow, pioneered for machine translation, became standard across natural language processing.

Reference-based comparison established a paradigm for evaluation. Rather than trying to define abstract rules for what makes good output, BLEU compared system outputs against what humans produce. This approach acknowledged that quality is best defined by example rather than by formal criteria. The same principle would later be applied to evaluating text generation, summarization, question answering, and eventually even large language models, where outputs are compared against human-written references or human preferences.

The focus on correlation with human judgments set a standard for validating automatic metrics. BLEU wasn't adopted because it perfectly captured every aspect of translation quality, but because it correlated well enough with human assessments to be useful. This established that automatic metrics should be judged by their ability to approximate human judgment rather than by theoretical elegance or mathematical sophistication. A simple metric that correlates well with humans is more valuable than a complex metric that doesn't.

Perhaps most subtly, BLEU introduced the idea that evaluation could itself be an object of research. The development of metrics like METEOR, ROUGE, and eventually BERTScore represented attempts to address BLEU's limitations while preserving its advantages. This created an ongoing research program around evaluation methodology, with papers proposing new metrics, analyzing their properties, and validating them against human judgments. Researchers treated evaluation as a critical challenge in its own right, alongside its use as a tool for measuring progress on other problems.