ROUGE Scores › Key Parameters

Michael BrenndoerferFebruary 23, 20264 min read
Language AI HandbookAll chapters
Part 53 of 67Evaluation Fundamentals
Chapter 408ROUGE Scores › Key Parameters
Reading section 9 of 13 · Key Parameters · 4 minSection 9 of 13 · 4 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 408 of 513

Section 9 of 13 · 4 min read

ROUGE Scores › Key ParametersOne section · Sections in this chapter9 / 13 ↓
Sections in this chapter13
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The key parameters for ROUGE score calculation are:

  • n: The n-gram order for ROUGE-N (typically 1 or 2). ROUGE-1 measures unigram overlap (content coverage), while ROUGE-2 measures bigram overlap (local word ordering). Higher orders (ROUGE-3, ROUGE-4) are rarely reported because they are too strict for typical summary lengths, often resulting in scores near zero due to the difficulty of matching long consecutive sequences.

  • beta: The balance parameter for ROUGE-L F-measure. When β>1\beta > 1, recall is weighted more heavily than precision. Standard implementations typically use β=1\beta = 1 (F1), though some favor recall with β=1.2\beta = 1.2. The choice depends on whether you want to penalize omissions or verbosity more severely.

  • use_stemmer: Whether to apply Porter stemming before matching. Stemming normalizes words to their root forms (e.g., "running" to "run"), improving robustness to morphological variation. Most modern implementations enable stemming by default to ensure that grammatical variations don't unfairly penalize otherwise correct summaries.

  • remove_stopwords: Whether to filter out common words ("the", "and", "is") before calculation. This focuses evaluation on content words but may discard important discourse markers. When reporting results, you should specify whether stopwords were removed to ensure comparability with other studies.

Out[13]:
Visualization
Line chart of ROUGE-L F-measure versus beta from 0.1 to 2.0 for two candidates, with vertical markers at beta 1 and 1.2.
Effect of the beta parameter on ROUGE-L F-measure for Candidate A (perfect precision, recall 0.889) and Candidate B (precision 0.889, recall 0.889). As beta increases, the F-measure weights recall more heavily, so Candidate A's score drops toward its recall value (0.889) while Candidate B's score remains constant. The vertical dashed lines mark beta equals 1 (standard F1) and beta equals 1.2 (recall-favoring).

The beta sensitivity plot reveals an important property of ROUGE-L: when a candidate has perfect precision (every word it contains is in the right order relative to the reference), higher beta values decrease its score by weighting the imperfect recall more heavily. When precision equals recall, the F-measure is constant across all beta values, as with Candidate B's symmetric scores.