ROUGE & METEOR (2004) › The Solution: Task-Specific and Semantically-Aware Evaluation

ROUGE and METEOR addressed BLEU's limitations through complementary approaches. ROUGE tackled the summarization problem by inverting BLEU's focus: instead of measuring precision (what fraction of generated content appears in references), ROUGE measured recall (what fraction of reference content appears in generated text). METEOR tackled the semantic equivalence problem by incorporating linguistic knowledge.synonym matching, stemming, and word order considerations.to recognize when different wordings conveyed the same meaning.

ROUGE recognized that summarization evaluation required different priorities than translation evaluation. While translation emphasized avoiding extraneous content (precision), summarization emphasized capturing important information (recall). ROUGE adapted BLEU's n-gram matching framework but inverted the perspective: instead of asking "what fraction of the generated summary's n-grams appear in references," ROUGE asked "what fraction of the reference summary's n-grams appear in the generated text."

The ROUGE family of metrics included multiple variants optimized for different aspects of summarization evaluation. ROUGE-N measured recall of n-grams of length N, with ROUGE-1 (unigram recall) and ROUGE-2 (bigram recall) being most commonly used. ROUGE-L measured longest common subsequence (LCS) between generated and reference summaries, capturing sentence-level structure. ROUGE-W measured weighted longest common subsequence, giving higher scores to sequences that were less fragmented. ROUGE-S measured skip-bigram co-occurrence statistics, allowing for flexible word ordering.

The mathematical formulation of ROUGE-N recall was straightforward: for each n-gram length N, ROUGE computed the recall as the number of matching n-grams between generated and reference summaries, divided by the total number of n-grams in the reference. This inverted BLEU's precision calculation, which divided matches by generated n-grams. The ROUGE-L variant added sentence-level structure by computing the longest common subsequence.the longest sequence of words that appeared in both summary and reference, not necessarily consecutively.then normalizing by reference length.

ROUGE's design philosophy reflected an important principle: evaluation metrics should align with task objectives. Since summarization's goal was information coverage, the metric should emphasize coverage. Since translation's goal included avoiding errors, metrics could emphasize precision. This task-specific design would become increasingly important as the field expanded into new domains.

METEOR addressed BLEU's semantic limitations by incorporating linguistic knowledge into the evaluation process. Instead of requiring exact word matches, METEOR could recognize semantic equivalence through synonym matching, morphological variations through stemming, and account for word order through alignment penalties. This enabled METEOR to give credit for meaning preservation even when word choices differed from references.

The core innovation was METEOR's multi-stage matching process. First, METEOR performed exact word matching, just like BLEU. Then it performed stemmed matching, where words were reduced to their roots (e.g., "running" and "runs" both become "run") and matches were counted. Finally, it performed synonym matching using WordNet, where words with the same or similar meanings were considered matches. This multi-stage approach ensured that semantic equivalence was recognized even when surface forms differed.

METEOR's scoring function balanced precision and recall rather than focusing solely on precision like BLEU. The harmonic mean of precision and recall.computed as F-score.provided a more balanced assessment. METEOR also introduced a fragmentation penalty based on word alignment: if matching words were scattered throughout the translation rather than aligned in phrases, METEOR applied a penalty. This encouraged translations to preserve phrase-level structure, not just individual word matches.

The alignment process was sophisticated. METEOR found the alignment between candidate and reference that maximized matches while minimizing fragmentation.the extent to which matched words were scattered rather than clustered. The fragmentation penalty was computed based on the number of "chunks" (contiguous sequences of matched words) relative to the total number of matches. More fragmented alignments received larger penalties, encouraging phrase-level preservation.

METEOR's formula combined these elements: after computing precision P and recall R from the multi-stage matching, METEOR computed F-score as the harmonic mean. Then it applied a fragmentation penalty based on the chunk-to-match ratio, resulting in: METEOR = (1 - penalty) × F-score. This formulation balanced semantic coverage (through recall), accuracy (through precision), and fluency (through the fragmentation penalty).