ROUGE Scores › Limitations and Impact

ROUGE has served as the de facto standard for summarization evaluation for two decades, letting rapid development and comparison of summarization systems. Its simplicity and computational efficiency have made large-scale benchmarking possible, driving progress from statistical methods to neural approaches. However, its limitations have become increasingly apparent as neural summarization models have grown more advanced and capable of generating novel phrasing rather than extracting sentences.

Like BLEU, ROUGE operates at the surface level of text. It cannot recognize paraphrases, synonyms, or semantic equivalence expressed with different words. A candidate summary stating "The feline consumed the rodent" receives zero ROUGE score against a reference saying "The cat ate the mouse," despite perfect semantic alignment. This limitation extends beyond simple synonyms to encompass entire paraphrased sentences. If a reference states "The company announced disappointing quarterly earnings," and a candidate reads "The firm revealed poor financial results for the past three months," ROUGE scores will be low despite the identical informational content.

This semantic blindness becomes particularly problematic when evaluating models trained on diverse corpora that have learned rich synonym sets. A language model might generate a perfectly accurate and readable summary that uses different vocabulary than the reference, simply because its training distribution favors different phrasing. Penalizing this stylistic variation as if it were an error conflates surface form with semantic content, creating misleading evaluation signals.

This limitation is particularly acute for abstractive summarization, where neural models generate novel phrasing rather than extracting sentences. Modern models often produce summaries that are fluent and accurate but receive low ROUGE scores due to lexical variation. As the field moves toward more abstractive systems, the disconnect between ROUGE scores and human judgment has widened, prompting the development of semantic evaluation metrics like BERTScore.

ROUGE rewards extractive behavior. A system that copies sentences verbatim from the source will score higher than one that skillfully paraphrases, even when paraphrasing produces better summaries. This has led to concerns that optimizing for ROUGE encourages extractive or "copy-and-paste" summarization rather than true understanding and condensation. Researchers have observed that models trained to maximize ROUGE often learn to select and concatenate sentences from the source document rather than synthesizing information, limiting the development of truly abstractive capabilities.

This extractive bias creates a perverse feedback loop in research. If ROUGE is the primary benchmark, systems that reach state-of-the-art scores by clever sentence selection will attract more attention and follow-up work than systems that reach lower scores through real abstractive synthesis. The benchmark thus shapes the research direction in ways that might not correspond to the ultimate goal of building systems that truly understand and condense information.

Research has shown that ROUGE scores correlate poorly with human judgments when evaluating abstractive summaries, though they remain reasonable for extractive summarization where word overlap is expected. A seminal study by Kryscinski et al. (2019) found that neural summarization models could reach high ROUGE scores while containing significant factual errors, because ROUGE cannot verify whether claims in the summary are supported by the source document. A summary might say "The company reported a 50 percent increase in profits" when the source says "a 15 percent increase," scoring well on ROUGE for capturing the right vocabulary but failing completely on the factual accuracy criterion that matters most for news summarization.

ROUGE scores tend to favor longer summaries. A verbose candidate that includes many words from the reference will score higher on recall than a concise summary that captures only the needed information. While precision metrics (available in ROUGE-L) help mitigate this, the dominant reporting of recall-oriented scores encourages redundancy. A candidate that repeats the same information multiple times or includes tangential details from the source can inflate ROUGE scores without improving utility.

Consider a reference summary of 50 words. A candidate that is 150 words long and includes every word from the reference plus additional detail will score 1.0 on ROUGE-N recall, even though it is three times the target length. Human judges would likely penalize this verbosity, but ROUGE rewards it. This is why length constraints are typically imposed during generation (using maximum token limits), since the metric itself gives no natural incentive for conciseness.

Some implementations address this by penalizing candidates materially longer than the reference, but this adds arbitrary thresholds not present in the original metric. Others report F-measure rather than pure recall to balance coverage against conciseness, though this still doesn't perfectly capture the human preference for information density.

ROUGE performance varies materially across domains. In technical domains with standardized terminology (like medical or legal summarization), ROUGE correlates reasonably well with quality because key terms are fixed. A medical summary mentioning "myocardial infarction" will likely match references using the same term, making lexical overlap a valid quality signal. In creative or subjective domains (like story summarization or opinion summarization), where multiple valid summaries might use completely different vocabulary, ROUGE gives little signal. A summary of a novel might focus on character development, plot, or themes, with virtually no word overlap between equally valid summaries.

This domain sensitivity means that ROUGE scores are not transferable across domains. A ROUGE-2 score of 0.25 on news summarization is good, while the same score on scientific paper summarization might be poor, and on creative writing summarization might be excellent. Researchers must always contextualize ROUGE scores with domain-specific baselines and human evaluation to know whether a given score is high or low quality.

Despite these limitations, ROUGE remains useful as a diagnostic tool and for tracking relative improvements in controlled settings. When a new model reaches higher ROUGE scores than a previous model on the same dataset with the same references, it likely is real progress, even if the absolute score doesn't indicate human-level quality. ROUGE is a necessary but insufficient condition for quality: low ROUGE almost certainly indicates a poor summary, while high ROUGE suggests the summary captures reference content, though it may still suffer from semantic errors or poor readability.

The research community has developed several approaches to complement ROUGE's limitations. BERTScore uses contextual embeddings to compute token-level similarity between candidate and reference, letting semantic matching even when vocabulary differs. BLEURT fine-tunes a BERT model on human judgment data, learning to predict quality scores that correlate better with human assessments. FactCC and similar factuality metrics specifically check whether claims in the summary are entailed by the source document, addressing the factual accuracy gap that ROUGE ignores entirely.

For more fine-grained evaluation, ROUGE is increasingly supplemented by:

• BERTScore, which uses contextual embeddings to measure semantic similarity rather than lexical overlap (covered in the next chapter)
• Human evaluation for fluency, coherence, and factual consistency
• Task-specific metrics like factuality checking for news summarization, which verify that claims in the summary are supported by the source document
• Questionnaire-based evaluation (like QAGS and QuestEval), which decompose summary quality into a set of yes/no questions about specific pieces of information

The field is moving toward multi-dimensional evaluation that separates content selection (what information is included) from surface realization (how it is expressed), recognizing that a single number cannot capture summary quality. ROUGE remains a cornerstone of this ecosystem. This gives a cheap, automatic signal for content coverage, while newer metrics handle semantic fidelity and human evaluation assesses ultimate utility.

An important practical note: despite its limitations, ROUGE's longevity as a benchmark has itself become a feature. Decades of published results use ROUGE, which makes it the lingua franca for comparing systems across time and across papers. Even if you personally prefer BERTScore, reporting ROUGE alongside it ensures your results are interpretable to the broader community and comparable to historical baselines. The social and scientific value of a common benchmark language should not be underestimated.