ROUGE Scores › Summary

ROUGE gives a family of recall-oriented metrics specifically designed for summarization evaluation. Unlike BLEU's precision-focused n-gram matching for translation, ROUGE measures how much of the reference content appears in the generated summary, prioritizing coverage over conciseness.

The four main variants serve different purposes:

• ROUGE-1 (unigram recall) captures content coverage at the word level. It verifies that the right vocabulary is present but cannot assess ordering or fluency. It is the most permissive metric and should always be paired with higher-order variants.
• ROUGE-2 (bigram recall) captures local word ordering. It penalizes summaries with the right words but scrambled sequences, which makes it a better indicator of linguistic quality and syntactic coherence.
• ROUGE-L uses longest common subsequence to capture sentence-level structure and word ordering without requiring consecutive matches. It balances precision and recall through F-measure calculation. This gives robustness to paraphrasing that reorganizes sentences while keeping key phrases.
• ROUGE-W applies superlinear weighting to consecutive matches, penalizing scattered word alignments and rewarding fluent, cohesive summaries. It is less commonly used due to the non-standardized weighting parameter.
• ROUGE-S measures skip-bigram co-occurrence, capturing ordered word pairs with arbitrary gaps. It occupies a middle ground between the strict locality of ROUGE-2 and the global flexibility of ROUGE-L.

Critical considerations for using ROUGE correctly:

• ROUGE is recall-oriented, which makes it suitable for summarization where coverage matters more than precision. This shows the task's priority of capturing needed information over avoiding extraneous content.
• Preprocessing choices (stemming, stopword removal, tokenization) materially impact scores and must be reported for reproducibility. Two results using different preprocessing are not directly comparable.
• Multiple references are handled by taking the maximum score across references, acknowledging the diversity of valid summaries for a single document.
• Modern neural summarizers often produce valid summaries that receive low ROUGE scores due to lexical variation. This shows the metric's limitations for abstractive evaluation.
• Domain context determines what constitutes a good ROUGE score. Always compare against domain-specific baselines, including the Lead-3 baseline for news.

While ROUGE has enabled rapid progress in summarization research, its reliance on surface-form matching limits its utility for evaluating abstractive systems. Current best practice combines ROUGE with semantic metrics like BERTScore and targeted human evaluation to capture the full picture of summary quality. This keeps automatic metrics serve their intended purpose without constraining the development of more advanced summarization capabilities.