ROUGE Scores › ROUGE-S: Skip-Bigram Overlap

Another variant worth understanding is ROUGE-S, which measures skip-bigram co-occurrence. Skip-bigrams are pairs of words that appear in the same order as the reference, but with arbitrary gaps between them. This concept bridges the gap between the strict locality of ROUGE-2 (consecutive words) and the global flexibility of ROUGE-L (any ordered subset).

Consider the reference: "The president spoke about climate change." A ROUGE-2 bigram is "spoke about," requiring consecutive words. A skip-bigram would include pairs like "president...climate" (words 2 and 5) or "The...change" (words 1 and 6), as long as the order is preserved. This captures semantic relationships between distant words that might be separated by modifiers or clauses.

ROUGE-S is computed similarly to ROUGE-2, but instead of counting consecutive bigrams, it counts ordered word pairs with any distance between them. A candidate summary that uses the same key words in the same order as the reference, even with different intervening words, receives credit for every matching pair. This gives a middle ground between ROUGE-1 (unordered) and ROUGE-2 (strictly consecutive).

The skip-bigram count for a candidate $C$ and reference $R$ is:

$$\text{Skip-bigram count} = |\{(w_i, w_j) : w_i \in C, w_j \in C, i < j, \exists (w_i, w_j) \in R\}|$$

where:

• $w_i, w_j$: words at positions $i$ and $j$ in the candidate text
• $C$: the candidate summary (as a sequence of words)
• $R$: the reference summary (as a sequence of words)
• $i < j$: the constraint that word $w_i$ appears before word $w_j$ in the candidate
• $\exists (w_i, w_j) \in R$: the condition that the same word pair appears in the reference in the same order

To compute the ROUGE-S score, the skip-bigram counts are normalized by the total number of skip-bigrams in the reference, yielding a recall measure analogous to ROUGE-N. Precision and F-measure can also be computed. A variant called ROUGE-SU combines skip-bigrams with unigrams, preventing degenerate cases where a very short candidate reaches high ROUGE-S by matching a small number of ordered pairs.

One practical issue with ROUGE-S is the skip distance parameter. In the original ROUGE formulation, skip-bigrams allow any distance, but this means very long documents can produce enormous numbers of candidate pairs. A document with 100 words generates roughly 4,950 potential skip-bigrams, compared to only 99 consecutive bigrams. Some implementations cap the maximum skip distance to prevent computational issues and to focus on semantically related word pairs rather than arbitrary long-range coincidences.

ROUGE-S is particularly useful for evaluating summaries that paraphrase by inserting words between key terms, though in practice, ROUGE-L often gives similar discriminative power with a more intuitive interpretation. As a result, ROUGE-S is less commonly reported than ROUGE-L in mainstream benchmarks, though it occasionally appears in studies specifically investigating paraphrase-reliable evaluation.