ROUGE Scores › ROUGE-N: N-gram Recall

The most straightforward variant, ROUGE-N, measures the proportion of n-grams in the reference summary that also appear in the generated summary. This is fundamentally a recall metric, unlike BLEU which computes precision. This distinction is significant: when evaluating summaries, we prioritize completeness over succinctness. A summary that mentions only one key fact from a ten-fact reference is precise (everything it says might be correct) but useless because it fails to deliver the bulk of the important information. ROUGE-N captures this completeness by checking how many chunks of the reference text appear in the candidate summary.

To build intuition before the formula, consider a simple scenario. A reference summary says "The economy grew strongly last quarter, driven by consumer spending and export growth." A generated summary says "The economy grew strongly." Both statements are true and every word in the candidate appears in the reference. But ROUGE-N would score this candidate poorly on recall because it captured only a small fraction of the reference's content. The metric correctly identifies that despite perfect precision, the candidate missed the explanation of why the economy grew. That "why" might be exactly what a reader needs.

ROUGE-N measures recall by computing the ratio of matching n-grams to total n-grams in the reference. For a specific n-gram order $n$, ROUGE-N is calculated as:

$$\text{ROUGE-N} = \frac{\sum_{S \in \text{References}} \sum_{\text{gram}_n \in S} \text{Count}_{\text{match}}(\text{gram}_n)}{\sum_{S \in \text{References}} \sum_{\text{gram}_n \in S} \text{Count}(\text{gram}_n)}$$

where:

• $\text{gram}_n$: an n-gram of order $n$ (a contiguous sequence of $n$ words)
• $\text{Count}_{\text{match}}(\text{gram}_n)$: the number of co-occurring n-grams in both the candidate and reference, capped by the reference count to prevent inflated scores from repetitive candidates
• $\text{Count}(\text{gram}_n)$: the total count of n-grams in the reference

To understand this formula intuitively, consider the numerator as a tally of every n-gram chunk found in the reference that also exists somewhere in the candidate summary. The denominator is the total number of n-gram chunks available in the reference. By dividing matches by total possible matches, we obtain a proportion between 0 and 1, where 1 indicates the candidate contains every n-gram from the reference (perfect recall) and 0 indicates no overlap whatsoever.

The capping mechanism in $\text{Count}_{\text{match}}$ deserves attention. If a candidate says "the cat the cat the cat" as a response to a reference saying "the cat sat on the mat," a naive count would give enormous credit for "the cat" matches. The cap ensures that if "the cat" appears once in the reference, it can only contribute one match regardless of how many times the candidate repeats it. This prevents a degenerate strategy of repeating common reference words to inflate scores.

When multiple reference summaries are available (common in summarization datasets, where different humans produce varying summaries of the same document), ROUGE-N typically takes the maximum score across all references:

$$\text{ROUGE-N}_{\text{multi}} = \max_i \text{ROUGE-N}(\text{candidate}, \text{reference}_i)$$

where:

• $i$: index over the set of reference summaries
• $\text{candidate}$: the generated summary being evaluated
• $\text{reference}_i$: the $i$-th human-written reference summary

This shows the intuition that a good summary only needs to match one valid human summary well, not all possible good summaries simultaneously. Because summarization is subjective, two equally valid summaries might focus on different aspects of a document or use completely different phrasing. Taking the maximum score acknowledges this diversity: if the candidate aligns well with any single reference, it likely captures valid content, even if it diverges from other references. This is analogous to grading an essay against the best matching rubric rather than requiring it to satisfy all rubrics simultaneously.

In practice, ROUGE-1 (unigram overlap) and ROUGE-2 (bigram overlap) are the most commonly reported metrics:

• ROUGE-1 measures content coverage at the word level. High ROUGE-1 indicates the summary captures the right vocabulary, but may not preserve word order or fluency. For example, "dog lazy the over jumps fox brown quick The" would reach perfect ROUGE-1 with "The quick brown fox jumps over the lazy dog" because it contains all the same words, despite being nonsensical. Thus, ROUGE-1 is a loose filter for content selection but cannot assess grammaticality or coherence. It answers the question: "Are the right words present somewhere in the summary?"

• ROUGE-2 measures local word ordering. High ROUGE-2 indicates the summary uses the right words and strings them together in ways that match the reference. In the previous example, the scrambled sentence would score zero on ROUGE-2 because no consecutive word pairs are preserved. The bigram "dog lazy" does not appear in the reference, and "lazy the" does not either. This makes ROUGE-2 a better indicator of linguistic quality and syntactic correctness. It answers: "Are the right words appearing together in the right local sequences?"

The relationship between ROUGE-1 and ROUGE-2 gives diagnostic information. If a summary scores high on ROUGE-1 but low on ROUGE-2, it likely has the right content but poor fluency or unusual ordering. If both scores are high, the summary captures content correctly and arranges it in sequences that resemble the reference. If both are low, the summary has missed the needed content entirely.

ROUGE-2 is generally considered a better indicator of summary quality than ROUGE-1 because it captures some syntactic structure. However, it is also stricter, often creating lower scores since bigram matches are rarer than unigram matches. In typical news summarization tasks, ROUGE-1 scores often range between 0.4 and 0.5 for strong systems, while ROUGE-2 scores might range between 0.2 and 0.3. You usually report both metrics together: ROUGE-1 to verify content coverage and ROUGE-2 to verify local coherence.

Higher-order ROUGE-N (ROUGE-3, ROUGE-4) is rarely reported in practice. The reason is statistical sparsity: as n grows, matching three or four consecutive words across a paraphrase becomes very difficult, and scores collapse toward zero for most real summaries. A ROUGE-4 score of 0.05 tells you very little about whether a summary is good. The useful range of n ends at 2 for most summarization tasks, though ROUGE-3 occasionally appears in tasks with very standardized output formats.