ROUGE Scores › ROUGE in Practice: Benchmarks and Correlation with Human Judgment

Michael BrenndoerferFebruary 23, 20263 min read
Language AI HandbookAll chapters
Part 53 of 67Evaluation Fundamentals
Chapter 408ROUGE Scores › ROUGE in Practice: Benchmarks and Correlation with Human Judgment
Reading section 10 of 13 · ROUGE in Practice: Benchmarks and Correlation with Human Judgment · 3 minSection 10 of 13 · 3 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 408 of 513

Section 10 of 13 · 3 min read

ROUGE Scores › ROUGE in Practice: Benchmarks and Correlation with Human JudgmentOne section · Sections in this chapter10 / 13 ↓
Sections in this chapter13
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Understanding ROUGE scores in isolation is less useful than understanding what they mean in the context of real benchmarks. The CNN/DailyMail dataset, introduced by Hermann et al. in 2015, has become the canonical testbed for news summarization. State-of-the-art neural systems on this dataset typically reach ROUGE-1 scores around 0.44-0.46, ROUGE-2 scores around 0.21-0.23, and ROUGE-L scores around 0.40-0.43. Earlier extractive systems from the 2000s achieved ROUGE-1 scores around 0.35-0.38, making the improvement from neural methods visually apparent as roughly 7-10 ROUGE-1 points over fifteen years of progress.

However, interpreting these numbers requires caution. A ROUGE-1 improvement of 1 point sounds modest but often is real progress when validated by human evaluation. Conversely, a 2-point improvement achieved through clever preprocessing or by exploiting dataset-specific patterns (such as including the first sentence of every article, which tends to be informative in news) may not generalize to other settings.

The correlation between ROUGE and human judgments has been extensively studied. Early work by Lin (2004) showed reasonable correlation for single-document news summarization, with Pearson correlations between ROUGE-1/ROUGE-2 and human readability and informativeness scores around 0.8-0.9 in controlled settings. However, later work by Callison-Burch et al. and others showed that this correlation degrades materially when:

  1. Systems use diverse paraphrasing (abstractive summaries)
  2. The domain differs from news (legal, scientific, creative texts)
  3. Reference summaries are not representative of the diversity of valid summaries
  4. System quality is already high and differences between systems are subtle

This degradation has motivated significant investment in alternative evaluation approaches, which we explore in the limitations section.

Out[14]:
Visualization
Horizontal bar chart showing ROUGE-1 score ranges for different system types from classical extractive to modern neural summarization.
Illustrative ROUGE-1 score ranges across different summarization system types and time periods, showing the progression from early extractive systems to modern neural models on the CNN/DailyMail benchmark. Neural abstractive systems have achieved higher ROUGE scores than classical extractive methods, though the gap has narrowed as both paradigms improved. Values are approximate and representative of published results.

The progression shown here illustrates an important point about ROUGE as a research signal. The Lead-3 baseline (simply taking the first three sentences of a news article) reaches surprisingly high ROUGE scores because news articles follow an inverted pyramid structure where the most important information appears first. This means that a ROUGE score below the Lead-3 baseline is a signal that something is wrong with the system, while scores above it indicate real learning of summarization patterns beyond position heuristics.