ROUGE Scores › Worked Example

Let's walk through a concrete example to see how these metrics differ. Consider a reference summary and two candidate summaries:

Reference: "The quick brown fox jumps over the lazy dog"

Candidate A: "The brown fox jumps over the lazy dog" (omits "quick")

Candidate B: "The quick fox jumps over the brown lazy dog" (reorders words)

These candidates illustrate two common failure modes in summarization: omission (missing content) and distortion (reordering content). Candidate A is like a summary that gets the structure right but misses one detail. Candidate B is like a summary that contains all the right words but rearranges them in a way that changes meaning. By calculating ROUGE scores manually, we can observe how different metrics penalize these errors differently.

For ROUGE-1, we count unigram matches:

Candidate A: 8 matches / 9 reference words = 0.889

Candidate B: 9 matches / 9 reference words = 1.000

ROUGE-1 considers Candidate B perfect despite the word reordering. The word "brown" moved from before "fox" to before "lazy dog," changing the phrase "quick brown fox" into "quick fox" and "brown lazy dog" into something somewhat different in meaning. But ROUGE-1 does not know or care about ordering. It simply checks that "brown" is present somewhere, which it is. This illustrates the core limitation of unigram matching: it captures presence but not arrangement. A summary could be completely scrambled yet reach perfect ROUGE-1, which is why higher-order n-grams or LCS-based measures are necessary.

For ROUGE-2, we need bigram matches. The reference bigrams are:

• "the quick", "quick brown", "brown fox", "fox jumps", "jumps over", "over the", "the lazy", "lazy dog"

Candidate A bigrams:

• "the brown", "brown fox", "fox jumps", "jumps over", "over the", "the lazy", "lazy dog"

Matches: "brown fox", "fox jumps", "jumps over", "over the", "the lazy", "lazy dog" = 6 matches

Score: 6 / 8 = 0.750

Candidate B bigrams:

• "the quick", "quick fox", "fox jumps", "jumps over", "over the", "the brown", "brown lazy", "lazy dog"

Matches: "the quick", "fox jumps", "jumps over", "over the", "lazy dog" = 5 matches

Score: 5 / 8 = 0.625

ROUGE-2 correctly penalizes Candidate B for the reordering, though it also penalizes Candidate A for omitting "quick." Notice how the insertion of "quick" before "fox" in Candidate B destroys the bigram "quick brown" (since "brown" is now elsewhere in the sentence) and replaces it with "quick fox" (not in reference). Simultaneously, "brown" now appears in the bigram "the brown" (not in reference) and "brown lazy" (not in reference), causing multiple bigram mismatches. ROUGE-2 captures this disruption effectively. This gives a more fine-grained quality signal than ROUGE-1.

For ROUGE-L, we find the longest common subsequence.

Reference: [The, quick, brown, fox, jumps, over, the, lazy, dog]

Candidate A: [The, brown, fox, jumps, over, the, lazy, dog]

The LCS is [The, brown, fox, jumps, over, the, lazy, dog] with length 8. Candidate A preserves all words except "quick" in the correct order, so the LCS simply skips "quick" from the reference side.

$$\begin{aligned} R_{\text{lcs}} &= \frac{8}{9} = 0.889 \\[6pt] P_{\text{lcs}} &= \frac{8}{8} = 1.0 \\[6pt] F_{\text{measure}} &= \frac{2 \cdot 0.889 \cdot 1.0}{0.889 + 1.0} = 0.941 \end{aligned}$$

Candidate A reaches perfect precision (every word in the candidate matches a word in the reference in the correct order) but imperfect recall (it missed "quick").

Candidate B: [The, quick, fox, jumps, over, the, brown, lazy, dog]

The LCS is [The, quick, fox, jumps, over, the, lazy, dog] with length 8. The word "brown" cannot be part of the LCS because of its position: in the reference, "brown" appears at position 3 (before "fox"), but in Candidate B, "brown" appears at position 7 (after "the"). The LCS algorithm must maintain relative order, so it cannot include "brown" from position 3 in the reference alongside "fox" from position 3 in the candidate (where "brown" has already been placed at position 7).

$$\begin{aligned} R_{\text{lcs}} &= \frac{8}{9} = 0.889 \\[6pt] P_{\text{lcs}} &= \frac{8}{9} = 0.889 \\[6pt] F_{\text{measure}} &= \frac{2 \cdot 0.889 \cdot 0.889}{0.889 + 0.889} = 0.889 \end{aligned}$$

So ROUGE-L gives Candidate A a higher score than Candidate B (0.941 vs 0.889), though both lose points for different reasons. Candidate A loses only recall (it omitted a word). Candidate B loses both recall and precision (it misplaced "brown," which counts as a gap in the LCS from both the reference side and the candidate side).

This example illustrates how different metrics capture different quality aspects, and why multiple ROUGE variants are typically reported together. ROUGE-1 alone would misleadingly suggest Candidate B is perfect. ROUGE-2 penalizes B heavily for reordering. ROUGE-L finds a middle ground, recognizing that B preserves some structure but is less coherent than A.