Summarization › Extractive Summarization

Extractive summarization frames the problem as a selection task: which sentences from the document are most important? This formulation avoids the harder generation problem entirely and guarantees that the output is grammatical (since the sentences come from a human-written source). The challenge shifts to scoring and ranking.

The simplest scoring approach treats sentences as bags of words and asks: which sentences contain the most representative vocabulary? This intuition leads to frequency-based methods, where words that appear often in the document are assumed to be important, and sentences that contain many such words score highly. While simple, this insight captures something real: documents are about their topic, and the topic is reflected in which words get used repeatedly. If a document about climate change uses "carbon dioxide," "emissions," and "temperature" throughout, then sentences containing those words are likely to be more central to the document's purpose than sentences about peripheral context.

The major limitation of frequency-based scoring is that it treats all individual sentences in isolation, without regard to how they relate to each other. A sentence that shares vocabulary with many other sentences is more central to the document's topic network than one that shares vocabulary with only a few. This observation motivates graph-based approaches that model sentence relationships explicitly.

Building on the classical text representation techniques we explored in Chapter 2, TF-IDF scoring applies directly to sentence extraction. In a multi-document summarization context, TF-IDF helps identify terms that are important within a document but not too common across all documents in the collection. For single-document summarization, a simpler frequency weighting suffices.

Let $f(w, s)$ denote the frequency of word $w$ in sentence $s$, and $F(w)$ denote the total frequency of word $w$ in the document. A basic sentence score is:

$$\text{score}(s) = \sum_{w \in s} \frac{f(w, s)}{F(w)}$$

where:

• $s$: the sentence being scored
• $w$: a word in sentence $s$
• $f(w, s)$: the count of word $w$ in sentence $s$
• $F(w)$: the total count of word $w$ across the entire document

This formulation gives higher scores to sentences containing words that are locally concentrated (high $f(w, s)$) but not ubiquitous throughout the document (moderate $F(w)$). The intuition is that words repeated heavily in one part of the document signal a local topic focus. Stopwords like "the" or "is" would have high $F(w)$ relative to $f(w, s)$, pulling their contribution toward zero and effectively filtering them out without requiring an explicit stopword list.

In practice, you preprocess text by removing stopwords and applying stemming or lemmatization before computing frequencies. This prevents common function words from dominating the scores and ensures that "running," "ran," and "runs" are treated as the same underlying concept. The resulting score approximates how much each sentence is "about" the main topic of the document, at least as measured by shared vocabulary.

The limitation of pure frequency scoring is that it treats all frequent words as equally informative. In a document about machine learning, "model" and "data" appear constantly and score well, but they carry little discriminative information. The TF-IDF adjustment in multi-document settings addresses this by downweighting terms that are common across many documents. For single-document summarization, you often need additional heuristics to distinguish informative sentences from those that merely restate background context. This gap between what is frequent and what is important motivates more sophisticated approaches.

A significant advancement over simple frequency counting is the graph-based approach, exemplified by TextRank (Mihalcea and Tarau, 2004). Rather than scoring sentences in isolation, TextRank treats the summarization problem as a graph ranking problem, similar in spirit to Google's PageRank algorithm for web pages.

The core idea is elegant: a sentence is important if it is similar to many other important sentences. Importance propagates through similarity relationships rather than being assigned independently to each sentence. This recursive definition captures something that frequency scoring misses: a sentence might not use many distinctive words, but if it is thematically central (similar to a large fraction of the document's other sentences), it is probably important. A sentence introducing a key concept that is then discussed throughout the rest of the document will share vocabulary with all those later sentences and therefore accumulate a high score even if it does not repeat distinctive terms internally.

The algorithm works in three steps. First, build a graph where each node is a sentence. Then add weighted edges between sentence pairs, where the edge weight captures their similarity. Finally, run a recursive ranking algorithm to assign importance scores, letting importance flow from sentence to sentence along similarity edges.

The similarity between two sentences $s_i$ and $s_j$ is typically measured using a normalized word-overlap formula. Let $W_i$ denote the set of unique words in sentence $s_i$ and $W_j$ the set of unique words in $s_j$. The similarity is:

$$\text{sim}(s_i, s_j) = \frac{|W_i \cap W_j|}{\log|W_i| + \log|W_j|}$$

where:

• $W_i \cap W_j$: the set of words appearing in both sentences
• $|W_i \cap W_j|$: the number of shared words (the intersection size)
• $\log|W_i|$ and $\log|W_j|$: length normalization terms that prevent longer sentences from automatically scoring higher simply because they contain more words

Without the log normalization in the denominator, a 50-word sentence and a 10-word sentence sharing 5 words would produce the same raw overlap as a 10-word sentence and a 10-word sentence sharing 5 words, even though the first pair is less semantically aligned. The logarithm applies a soft penalty that grows with sentence length but does not grow as fast as the raw word count would. This normalization is important in practice because documents often contain sentences of very different lengths, and naive overlap would systematically favor the longer ones.

The ranking score is then computed iteratively. Let $\text{WS}(s_i)$ denote the score assigned to sentence $s_i$. Starting from a uniform initialization, scores are updated according to:

$$\text{WS}(s_i) = (1 - d) + d \cdot \sum_{s_j \in \text{In}(s_i)} \frac{w_{ji}}{\sum_{s_k \in \text{Out}(s_j)} w_{jk}} \cdot \text{WS}(s_j)$$

where:

• $d$: a damping factor (typically $d = 0.85$), representing the probability that a random traversal follows a similarity link rather than jumping uniformly to a random sentence
• $\text{In}(s_i)$: the set of sentences that link to $s_i$ (all other sentences in a fully connected graph)
• $w_{ji}$: the weight of the directed edge from $s_j$ to $s_i$, equal to $\text{sim}(s_j, s_i)$
• $\sum_{s_k \in \text{Out}(s_j)} w_{jk}$: the sum of all edge weights leaving $s_j$, used to normalize so that outgoing weights sum to 1

The term $(1 - d)$ provides a uniform base score, preventing sentences with no incoming edges from scoring zero. This mirrors the "teleportation" term in PageRank. The iteration continues until scores converge, typically within 30 to 50 steps. Once scores stabilize, sentences are ranked in descending order and the top $k$ are selected.

The convergence property of this iteration is guaranteed by the Perron-Frobenius theorem: because the graph is fully connected (every sentence has an edge to every other sentence, even if the weight is very small) and the damping factor keeps a non-zero base probability, the transition matrix is ergodic and the power iteration converges to a unique stationary distribution. In practice, you can check convergence by measuring the maximum absolute difference between successive score vectors and stopping when this falls below a small threshold like $10^{-6}$.

A key strength of TextRank over frequency-based methods is that it does not require any external knowledge or training data. It is fully unsupervised and works on any domain. This makes it useful for documents in specialized areas where training data is scarce: medical records, legal documents, or internal corporate reports. It is also fast, running in time proportional to the square of the number of sentences, which is manageable for documents of typical length.

Not all extractive approaches rely on content analysis. A surprisingly effective heuristic is position: sentences near the beginning of a document are often more important than sentences in the middle, because good writing typically front-loads key information.

In news articles, the "inverted pyramid" structure ensures the most important information appears first: the first paragraph answers who, what, when, where, and why, and subsequent paragraphs add detail in decreasing order of importance. This structure evolved from practical necessity: when newspapers had to cut articles from the bottom to fit available space, the most important content needed to survive truncation. An automatic summarizer that knows this convention can simply take the first one or two sentences of a news article and perform surprisingly well.

In scientific papers, the abstract, introduction, and conclusion sentences carry disproportionate importance. In long-form essays, section opening sentences often summarize what follows. In structured documents like annual reports, the executive summary and key metrics paragraphs carry most of the actionable content. These structural patterns are not accidental. They reflect conventions that writers adopt to help readers move through long documents. An automatic summarizer that ignores these conventions is throwing away strong signal that comes for free.

A common approach combines position scores with content scores using a weighted sum. If $\text{content\_score}(s)$ is a frequency or graph-based importance score and $\text{position\_score}(s)$ captures where the sentence appears (typically $1/\text{rank}(s)$ where the rank is 1 for the first sentence), the final score is:

$$\text{final\_score}(s) = \alpha \cdot \text{content\_score}(s) + (1 - \alpha) \cdot \text{position\_score}(s)$$

where:

• $\alpha \in [0, 1]$: a mixing weight controlling the balance between content and position signals
• $\text{content\_score}(s)$: frequency or graph-based importance score
• $\text{position\_score}(s)$: typically $1/\text{rank}(s)$ where $\text{rank}(s)$ is the sentence's position in the document (1-indexed)

The optimal $\alpha$ varies by document type. News articles benefit from heavy position weighting because their structure is highly regular. Research papers benefit from content weighting with position bonuses for certain structural positions (abstract sentences, section headings). For highly conversational or unstructured text, position provides almost no useful signal and $\alpha$ should be close to 1. Choosing $\alpha$ is itself a modeling decision that can be tuned if labeled data is available, or set heuristically based on domain knowledge.

A naive sentence ranking approach has a critical flaw: it will often select multiple sentences that say essentially the same thing. The top-scored sentences in a document about climate change might all be variants of "global temperatures are rising," producing a summary that is accurate but uninformative as a whole. Readers benefit much more from a summary that covers different aspects of the topic than from one that repeats the most prominent theme three times.

This problem is especially acute for TextRank. Because the algorithm assigns high scores to sentences that are similar to many other sentences, it naturally tends to select sentences that are all similar to each other. The most "central" sentences in a document cluster around the main theme. Selecting the top three by score can give you three near-paraphrases of the same core idea. You end up with a summary that is highly faithful to the main topic but completely fails to convey the document's breadth.

This motivates Maximum Marginal Relevance (MMR), a selection criterion that balances relevance against redundancy. When selecting the next sentence to add to the summary, MMR scores each remaining candidate $s$ as:

$$\text{MMR}(s) = \lambda \cdot \text{Sim}_1(s, Q) - (1 - \lambda) \cdot \max_{s_j \in S} \text{Sim}_2(s, s_j)$$

where:

• $Q$: the query or topic of interest (the document as a whole in generic summarization)
• $S$: the set of sentences already selected for the summary
• $\text{Sim}_1(s, Q)$: similarity of candidate $s$ to the query, measuring relevance
• $\text{Sim}_2(s, s_j)$: similarity of candidate $s$ to an already-selected sentence $s_j$, measuring redundancy
• $\lambda \in [0, 1]$: a parameter controlling the relevance-redundancy tradeoff; $\lambda = 1$ gives pure relevance ranking, $\lambda = 0$ gives pure diversity

The greedy MMR algorithm selects the highest-scoring sentence under this criterion, adds it to $S$, recomputes scores for remaining candidates (since the redundancy term now includes the newly added sentence), and repeats until the desired summary length is reached.

This algorithm is greedy. Because MMR selects one sentence at a time and updates the redundancy calculation after each selection, the chosen sentences depend on the order of selection. The first sentence chosen is the most relevant to the query (no penalty for redundancy yet). The second sentence is the one that best balances relevance with distinctiveness from the first. This sequential process generally produces better coverage than selecting all $k$ sentences simultaneously by pure relevance, because each new selection is made with full awareness of what has already been included.

Classical extractive methods treat sentence scoring as an independent classification or ranking problem, but neural extractive models learn richer sentence representations that capture contextual meaning rather than just surface word overlap.

Neural extractive summarizers typically work in two stages. First, a pre-trained encoder (such as BERT or a similar transformer model) processes the entire document and produces contextual embeddings for each sentence. Unlike TF-IDF or word overlap features, these embeddings capture semantic meaning: "automobile" and "car" would produce similar embeddings even though they share no vocabulary. Second, a classification head predicts whether each sentence should be included in the summary, trained on labeled datasets where human annotators marked summary-worthy sentences.

The key advantage over classical methods is that the encoder understands language rather than merely counting words. A sentence might be identified as important because it introduces a concept (even if that concept uses unusual vocabulary), because it contains a causal claim that other sentences elaborate on, or because it appears to be a thesis statement even in mid-document position.

Models like BertSum (Liu and Lapata, 2019) demonstrated that fine-tuning BERT for extractive summarization substantially outperforms classical methods on standard benchmarks. The model adds a special [CLS] token at the beginning of each sentence, and the representation of this token after BERT encoding is used as the sentence-level feature for the classification decision. By training jointly over all sentences in the document, the model can also capture cross-sentence dependencies that sentence-by-sentence scoring cannot.

The main limitation of neural extractive approaches is that they require labeled training data: you need documents where human annotators have identified which sentences are summary-worthy. For common domains like news, such datasets exist. For specialized domains, creating this annotation data is expensive and time-consuming. Classical methods like TextRank remain valuable precisely because they require no labeled data at all.