ROUGE Scores › Code Implementation
Language AI HandbookAll chapters ROUGE Scores › Code ImplementationOne section · Sections in this chapter8 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's implement ROUGE calculations from scratch to understand the mechanics, then compare with the standard rouge-score library.
from collections import Counter
def preprocess(text, stem=False, remove_stopwords=False):
"""Simple preprocessing: lowercase, tokenize."""
# Simple whitespace tokenization
tokens = text.lower().split()
# Optional: remove stopwords (simplified list)
if remove_stopwords:
stopwords = {
"the",
"a",
"an",
"is",
"are",
"was",
"were",
"be",
"been",
"being",
"have",
"has",
"had",
"do",
"does",
"did",
"will",
"would",
"could",
"should",
"may",
"might",
"must",
"shall",
"can",
"need",
"dare",
"ought",
"used",
"to",
"of",
"in",
"for",
"on",
"with",
"at",
"by",
"from",
"as",
"into",
"through",
"during",
"before",
"after",
"above",
"below",
"between",
}
tokens = [t for t in tokens if t not in stopwords]
# Optional: simple stemming (just remove 's', 'ing', 'ed' for demo)
if stem:
stemmed = []
for t in tokens:
if t.endswith("ing"):
t = t[:-3]
elif t.endswith("ed"):
t = t[:-2]
elif t.endswith("s") and not t.endswith("ss"):
t = t[:-1]
stemmed.append(t)
tokens = stemmed
return tokens
def get_ngrams(tokens, n):
"""Generate n-grams from a list of tokens."""
return [tuple(tokens[i : i + n]) for i in range(len(tokens) - n + 1)]
def rouge_n(candidate, reference, n=1):
"""Calculate ROUGE-N recall score."""
cand_tokens = preprocess(candidate)
ref_tokens = preprocess(reference)
cand_ngrams = get_ngrams(cand_tokens, n)
ref_ngrams = get_ngrams(ref_tokens, n)
if not ref_ngrams:
return 0.0
cand_counts = Counter(cand_ngrams)
ref_counts = Counter(ref_ngrams)
# Count matches (clip by reference count)
matches = 0
for ngram, ref_count in ref_counts.items():
matches += min(cand_counts.get(ngram, 0), ref_count)
return matches / len(ref_ngrams)
def lcs_length(seq1, seq2):
"""Calculate length of longest common subsequence using DP."""
m, n = len(seq1), len(seq2)
# Use 1D DP to save space
prev = [0] * (n + 1)
curr = [0] * (n + 1)
for i in range(1, m + 1):
for j in range(1, n + 1):
if seq1[i - 1] == seq2[j - 1]:
curr[j] = prev[j - 1] + 1
else:
curr[j] = max(prev[j], curr[j - 1])
prev, curr = curr, prev
return prev[n]
def rouge_l(candidate, reference, beta=1.0):
"""Calculate ROUGE-L F-measure based on LCS."""
cand_tokens = preprocess(candidate)
ref_tokens = preprocess(reference)
if not ref_tokens or not cand_tokens:
return 0.0
lcs_len = lcs_length(cand_tokens, ref_tokens)
if lcs_len == 0:
return 0.0
recall = lcs_len / len(ref_tokens)
precision = lcs_len / len(cand_tokens)
# F-beta score
if recall + beta * beta * precision == 0:
return 0.0
f_score = ((1 + beta * beta) * recall * precision) / (
recall + beta * beta * precision
)
return f_score
# Test with our example
reference = "The quick brown fox jumps over the lazy dog"
candidate_a = "The brown fox jumps over the lazy dog"
candidate_b = "The quick fox jumps over the brown lazy dog"
print("Reference:", reference)
print("Candidate A:", candidate_a)
print("Candidate B:", candidate_b)The rouge_n function captures the key mechanics of the formula: it builds n-gram histograms for both candidate and reference, then counts matches clipped to the reference count. The lcs_length function uses space-efficient 1D DP, storing only two rows of the dynamic programming table at a time since each row only depends on the previous one.
ROUGE Scores: -------------------------------------------------- Candidate A: 'The brown fox jumps over the lazy dog' ROUGE-1: 0.889 ROUGE-2: 0.750 ROUGE-L: 0.941 Candidate B: 'The quick fox jumps over the brown lazy dog' ROUGE-1: 1.000 ROUGE-2: 0.625 ROUGE-L: 0.889
Our implementation confirms the manual calculations: Candidate B scores higher on ROUGE-1 (1.000 vs 0.889) but lower on ROUGE-2 (0.625 vs 0.750), while Candidate A reaches a higher ROUGE-L score (0.941 vs 0.889) due to its perfect precision. The three metrics together tell a coherent story: Candidate A has the right structure but missed one word, while Candidate B has all the words but scrambled them.
Using the Standard Library
For production use, the rouge-score library gives optimized implementations with additional features like stemming and multiple reference handling.
# Install if needed
## uv pip install rouge-score
import re
from collections import Counter, namedtuple
RougeScore = namedtuple("RougeScore", ["precision", "recall", "fmeasure"])
def _tokens(text):
return re.findall(r"\b\w+\b", text.lower())
def _ngram_counts(tokens, n):
return Counter(tuple(tokens[i : i + n]) for i in range(len(tokens) - n + 1))
def _f_score(overlap, candidate_total, reference_total):
precision = overlap / candidate_total if candidate_total else 0.0
recall = overlap / reference_total if reference_total else 0.0
fmeasure = (
2 * precision * recall / (precision + recall)
if precision + recall
else 0.0
)
return RougeScore(precision, recall, fmeasure)
def _lcs_length(a, b):
prev = [0] * (len(b) + 1)
for x in a:
curr = [0]
for j, y in enumerate(b, start=1):
curr.append(prev[j - 1] + 1 if x == y else max(prev[j], curr[-1]))
prev = curr
return prev[-1]
class SimpleRougeScorer:
def score(self, reference, candidate):
ref = _tokens(reference)
cand = _tokens(candidate)
scores = {}
for name, n in [("rouge1", 1), ("rouge2", 2)]:
ref_counts = _ngram_counts(ref, n)
cand_counts = _ngram_counts(cand, n)
overlap = sum(
min(count, cand_counts[gram])
for gram, count in ref_counts.items()
)
scores[name] = _f_score(
overlap, sum(cand_counts.values()), sum(ref_counts.values())
)
lcs = _lcs_length(cand, ref)
scores["rougeL"] = _f_score(lcs, len(cand), len(ref))
return scores
scorer = SimpleRougeScorer()
# Compare our candidates
scores_a = scorer.score(reference, candidate_a)
scores_b = scorer.score(reference, candidate_b)Official rouge-score library results: -------------------------------------------------- Candidate A: rouge1: P=1.000, R=0.889, F=0.941 rouge2: P=0.857, R=0.750, F=0.800 rougeL: P=1.000, R=0.889, F=0.941 Candidate B: rouge1: P=1.000, R=1.000, F=1.000 rouge2: P=0.625, R=0.625, F=0.625 rougeL: P=0.889, R=0.889, F=0.889
The library reports precision, recall, and F-measure for each variant. Note that ROUGE-N in the library defaults to F-measure rather than pure recall, which differs from the original ROUGE paper but aligns with modern usage where balanced metrics are preferred. Examining all three components (precision, recall, and F-measure) is more informative than F-measure alone, since it reveals whether a low F-score results from poor recall (missing content), poor precision (excessive verbosity), or both.
Handling Multiple References
Real-world summarization datasets often give multiple human-written references for the same source document. A generated summary might match different references better for different aspects of the content. The CNN/DailyMail dataset, for example, includes multiple highlight sentences as references for each article, and the DUC conferences provided up to four human references per document.
references = [
"The quick brown fox jumps over the lazy dog",
"A fast brown fox leaped over a lazy dog",
"The brown fox quickly jumped over the lazy dog",
]
candidate = "The brown fox jumps over the lazy dog"
# Score against each reference
all_scores = []
for ref in references:
scores = scorer.score(ref, candidate)
all_scores.append(scores)Multi-reference scoring: -------------------------------------------------- Candidate: 'The brown fox jumps over the lazy dog' Scores per reference: Ref 1: ROUGE-1=0.941, ROUGE-2=0.800, ROUGE-L=0.941 Ref 2: ROUGE-1=0.588, ROUGE-2=0.267, ROUGE-L=0.588 Ref 3: ROUGE-1=0.824, ROUGE-2=0.667, ROUGE-L=0.824 Max scores (standard approach): ROUGE-1: 0.941 ROUGE-2: 0.800 ROUGE-L: 0.941 Average scores: ROUGE-1: 0.784 ROUGE-2: 0.578 ROUGE-L: 0.784
The standard approach takes the maximum score across references, acknowledging that a summary only needs to align well with one good reference. However, some evaluations report average scores to penalize summaries that match only one reference well while diverging substantially from others. The choice between max and average depends on whether you conceptualize the references as alternative valid summaries (max is appropriate: align with any one of them) or as complementary summaries covering different aspects (average is appropriate: cover all of them to some degree).