Summarization › Summarization Evaluation

Evaluating summaries is difficult. Unlike translation or question answering, there is rarely a single correct answer. A good summary of a 10,000-word article could take many valid forms, using different sentences, different phrasings, and covering different subsets of the content. This ambiguity makes automatic evaluation challenging and human evaluation expensive.

The challenge is compounded by the multiplicity of valid summaries. Two equally skilled human summarizers given the same document will produce summaries that differ substantially in vocabulary and structure, yet both may be excellent. An automatic metric that rewards similarity to one reference summary will penalize the other, even if both are equally good. This multi-reference problem is a fundamental tension in summarization evaluation that no current metric fully resolves.

A further complication is that different evaluation criteria matter differently in different applications. A medical summary used to brief a clinician needs extreme faithfulness: a single hallucinated drug dosage could harm a patient. A news summary for a casual reader can afford some paraphrasing that slightly stretches the original claims. An internal corporate summary that will be read by domain experts can use technical vocabulary freely; one targeted at executives may need to simplify. Any single automatic metric that treats these scenarios identically will be misleading in at least one of them.

The dominant automatic evaluation metric for summarization is ROUGE (Recall-Oriented Understudy for Gisting Evaluation), introduced by Lin (2004). ROUGE measures the overlap between a generated summary and one or more reference summaries written by humans.

ROUGE-N measures n-gram overlap. The recall formulation counts how many of the reference n-grams appear in the candidate summary:

$$\text{ROUGE-N}_{R} = \frac{\sum_{S \in \text{Refs}} \sum_{\text{gram}_n \in S} \text{Count\_match}(\text{gram}_n)}{\sum_{S \in \text{Refs}} \sum_{\text{gram}_n \in S} \text{Count}(\text{gram}_n)}$$

where:

• $\text{Refs}$: the set of reference summaries
• $\text{gram}_n$: each n-gram (sequence of $n$ consecutive tokens) in a reference summary
• $\text{Count\_match}(\text{gram}_n)$: the count of $\text{gram}_n$ in the candidate summary, clipped to the count in the reference
• $\text{Count}(\text{gram}_n)$: the count of $\text{gram}_n$ in the reference summary

This is a recall-oriented measure: it rewards summaries that cover the reference content. Precision measures the complementary direction (how much of the candidate is in the reference), and the F1 harmonic mean balances both:

$$\text{ROUGE-N}_{F_1} = \frac{2 \cdot P \cdot R}{P + R}$$

where $P$ is precision (fraction of candidate n-grams found in the reference) and $R$ is recall (fraction of reference n-grams found in the candidate).

ROUGE-1 uses unigram overlap and measures vocabulary coverage broadly. ROUGE-2 uses bigram overlap and is more sensitive to phrasing: two summaries covering the same facts in different words will score well on ROUGE-1 but potentially poorly on ROUGE-2 if they use different collocations. This is why ROUGE-2 is often a better discriminator between system quality levels, though it also penalizes valid paraphrases more harshly. In general, ROUGE-1 tells you whether the candidate covers the right topics; ROUGE-2 tells you whether it also uses similar phrasing to the reference.

ROUGE-L measures the longest common subsequence (LCS) between the candidate and reference, capturing structural similarity beyond exact n-gram matches. The recall-based formulation is:

$$\text{ROUGE-L}_{R} = \frac{\text{LCS}(C, R)}{|R|}$$

where:

• $C$: the candidate summary (as a token sequence)
• $R$: the reference summary (as a token sequence)
• $\text{LCS}(C, R)$: the length (in tokens) of the longest common subsequence of $C$ and $R$
• $|R|$: the number of tokens in the reference summary

Unlike ROUGE-N, ROUGE-L does not require the matching tokens to be contiguous. A candidate that captures the same sequence of ideas as the reference but skips connecting words can still score well under ROUGE-L. This makes it somewhat more robust to paraphrasing than ROUGE-2, though less sensitive to local phrasing choices. ROUGE-L is often used as a single composite metric when you want something in between the n-gram granularity of ROUGE-1 and ROUGE-2.

BERTScore (Zhang et al., 2019) addresses ROUGE's vocabulary limitation by using contextual embeddings instead of exact token matches. For each token in the candidate summary, BERTScore finds the most similar token in the reference according to their BERT embeddings, and vice versa.

The precision component measures the average similarity of each candidate token to its closest reference match:

$$P_{\text{BERT}} = \frac{1}{|C|} \sum_{c_i \in C} \max_{r_j \in R} \cos(\mathbf{e}_{c_i}, \mathbf{e}_{r_j})$$

The recall component measures the average similarity of each reference token to its closest candidate match:

$$R_{\text{BERT}} = \frac{1}{|R|} \sum_{r_j \in R} \max_{c_i \in C} \cos(\mathbf{e}_{r_j}, \mathbf{e}_{c_i})$$

where:

• $C$: the set of candidate summary tokens
• $R$: the set of reference summary tokens
• $\mathbf{e}_{c_i}$: the contextual embedding of candidate token $c_i$ from BERT (a vector in $\mathbb{R}^d$)
• $\mathbf{e}_{r_j}$: the contextual embedding of reference token $r_j$ from BERT
• $\cos(\mathbf{u}, \mathbf{v}) = \frac{\mathbf{u} \cdot \mathbf{v}}{\|\mathbf{u}\| \|\mathbf{v}\|}$: cosine similarity between two embedding vectors

The BERTScore F1 combines precision and recall in the standard way. BERTScore better captures semantic equivalence: "vehicle" and "car" would contribute positive similarity even though they never match under ROUGE-N. BERTScore correlates better with human judgments than ROUGE in most evaluations, but it comes with higher computational cost (computing BERT embeddings for every token in every summary and reference) and the additional complexity of choosing which BERT layer and model to use. The choice of BERT model matters because different models produce embeddings with different geometric properties, and the optimal choice varies by domain.

When human evaluation is conducted (typically for research publications or high-stakes applications), evaluators assess multiple dimensions of summary quality:

• Faithfulness: Does the summary contain only information that can be supported by the source? Unfaithful summaries contain hallucinated facts. This is often the most critical dimension for production applications because users who discover a single false claim may lose trust in the entire system.
• Relevance: Does the summary cover the most important aspects of the source? A highly faithful but irrelevant summary copies trivial details while omitting the key findings. Relevance is subjective and depends on what the reader needs.
• Coherence: Is the summary well-organized and logical? Extracted sentences may be individually accurate but collectively incoherent if they come from different parts of the document with no narrative thread connecting them.
• Fluency: Is the language grammatically correct and natural? Abstractive models sometimes produce disfluent text near their generation length limits, or when copying rare vocabulary from the source.

These dimensions often trade off against each other. Extractive systems score very high on faithfulness (since they copy) but sometimes lower on relevance and coherence. Abstractive systems can score higher on fluency and relevance but are more likely to introduce factual errors. Instruction-tuned LLMs generally score well on fluency and can be guided to prioritize faithfulness with appropriate prompting, but they require careful evaluation in high-stakes settings.

Human evaluation typically uses rating scales (1-5 or 1-7) on each dimension, administered through crowdsourcing platforms or expert panels. The cost ranges from a few cents per judgment on crowdsourcing platforms to hundreds of dollars per page for expert legal or medical review. This cost is why automatic metrics remain dominant in research, even though they are imperfect. For production systems, a practical approach is to use automatic metrics for rapid iteration during development, then conduct targeted human evaluation on a representative sample before deployment.