ROUGE Scores › Preprocessing and Configuration

ROUGE scores are sensitive to preprocessing choices. Standard implementations include several configurable options that materially impact scores. Understanding these choices is needed for fair comparison and reproducibility, as a ROUGE score reported without preprocessing details is in effect meaningless.

This might seem like a minor technical point, but it has caused real confusion in the research literature. Two papers might report ROUGE-1 scores for systems on the same dataset, but if one uses stemming and the other does not, the scores are not directly comparable. The same system can exhibit a several-point difference in ROUGE scores depending solely on whether stopwords were removed. Careful researchers always specify their preprocessing pipeline, and careful readers always check before drawing conclusions from score comparisons.

Most ROUGE implementations apply Porter stemming or similar algorithms to normalize words to their root forms. This allows "running," "runs," and "ran" to match, increasing robustness to morphological variation. Stemming collapses different inflections of the same word into a common token, recognizing that "the cat runs" and "the cat ran" convey the same core information despite tense differences.

Without stemming, "children play" and "child plays" would score poorly despite semantic equivalence. With stemming, both become "child play," creating perfect matches. However, stemming can occasionally over-collapse distinct meanings. For example, "universe" and "universal" might collapse to the same stem despite different meanings, and "organization" and "organism" might partially overlap in their stems. Such errors are rare enough that stemming generally improves evaluation reliability, but they are worth keeping in mind when examining cases where ROUGE scores seem anomalous.

The Porter stemmer applies a sequence of suffix stripping rules. Words ending in "-ing" have that suffix removed (with additional rules for minimal stem length). Words ending in "-tion" become "-t". The algorithm is entirely rule-based, which makes it fast but imperfect. For specialized domains with unusual morphology, domain-specific stemmers or lemmatizers might work better, though the research community predominantly uses Porter stemming for consistency.

Optionally, stopwords (common words like "the," "and," "is") can be removed before computing ROUGE. This focuses the metric on content words rather than function words, theoretically making the evaluation more sensitive to semantic substance rather than syntactic scaffolding.

However, stopword removal is controversial in summarization evaluation:

• Pro: Eliminates inflated scores from common phrases like "in the" or "of the" that appear in virtually every English sentence. Without removal, a system could reach moderate ROUGE scores simply by generating grammatical English sentences with little relation to the source content.
• Con: Removes important discourse markers and can make grammaticality harder to assess. Stopwords carry relational meaning (prepositions indicate spatial and temporal relationships; conjunctions indicate logical connections) that can be important for accurate summarization.

Most standard evaluations (like the CNN/DailyMail benchmark) report ROUGE scores both with and without stopword removal, letting researchers to assess both content selection and linguistic fluency. When you report ROUGE without specifying this choice, readers cannot reproduce your results or make fair comparisons.

ROUGE requires consistent tokenization between candidate and reference. Unlike BLEU which often uses specialized tokenizers, ROUGE typically uses simple whitespace tokenization or the same tokenizer used for the task. Inconsistent tokenization (e.g., treating "don't" as one word vs. "do n't" as two) can materially alter scores.

Punctuation handling also matters. Some implementations strip all punctuation, while others treat it as separate tokens. If one system outputs "However, the plan failed" and another outputs "However the plan failed," punctuation-sensitive tokenization will yield different ROUGE scores despite identical content words. When comparing systems or recreating results, ensure all candidates and references use the same tokenization pipeline.

A related subtlety involves case normalization. Most implementations lowercase all text before computing ROUGE, which is sensible since "President" and "president" refer to the same entity. However, for tasks where capitalization carries semantic meaning (named entity recognition, for instance), case-sensitive evaluation might be more appropriate. Again, the key is consistency and transparency.

A reliable preprocessing pipeline for ROUGE evaluation proceeds in a consistent order:

1. Lowercase all text to remove spurious case differences
2. Apply tokenization (whitespace-split or a dedicated tokenizer)
3. Strip punctuation tokens or normalize punctuation to whitespace
4. Optionally remove stopwords from the token list
5. Optionally apply stemming to remaining tokens
6. Compute ROUGE on the resulting token sequences

Every step in this pipeline must be identical for candidate and reference texts. Any asymmetry, such as stemming one but not the other, will produce scores that don't reflect actual overlap.