Summarization › Abstractive Summarization

Abstractive summarization is fundamentally different from extraction. Instead of copying sentences, an abstractive system reads the source and generates a new text from scratch. This allows the summary to paraphrase, combine ideas from multiple sentences, omit irrelevant details, and express concepts in cleaner or more concise language than the original.

The cost of this flexibility is substantial technical complexity. An abstractive summarizer must do what extractive systems do not: understand content deeply enough to paraphrase it, decide what to omit without losing meaning, and produce fluent natural language. This is a generation problem, requiring all the machinery of language modeling we have built up in earlier parts of this handbook.

The practical implications are significant. Extractive systems can be deployed with no training data and run in milliseconds on a laptop. Abstractive systems require gigabytes of training data, substantial compute for training, and inference that is orders of magnitude slower. But when they work well, abstractive summaries are qualitatively superior: more concise, more fluent, and more informative than any extractive alternative could produce. An abstractive system can fuse facts from three separate paragraphs into a single tight sentence, something extraction can never achieve.

The dominant framework for abstractive summarization is sequence-to-sequence (seq2seq) learning, which we covered in depth in Chapter 9. A seq2seq model consists of an encoder that reads the source document and produces a latent representation, and a decoder that generates the summary token by token conditioned on that representation.

For summarization, the encoder processes the full input document. Each input token $x_i$ is transformed into a hidden state $h_i$ as part of the full encoding:

$$(h_1, h_2, \ldots, h_n) = \text{Encoder}(x_1, x_2, \ldots, x_n)$$

where:

• $x_1, x_2, \ldots, x_n$: the sequence of input tokens (the source document)
• $h_i$: the hidden state corresponding to input token $x_i$, encoding contextual information about $x_i$ in the context of all other tokens
• $n$: the number of input tokens

The decoder then generates summary tokens autoregressively. At each time step $t$, it predicts the next token $y_t$ conditioned on all encoder hidden states and all previously generated tokens:

$$P(y_t \mid y_1, \ldots, y_{t-1}, x_1, \ldots, x_n) = \text{softmax}\bigl(W_o \cdot d_t\bigr)$$

where:

• $d_t = \text{Decoder}(h_1, \ldots, h_n, y_1, \ldots, y_{t-1})$: the decoder hidden state at time step $t$
• $W_o$: the output projection matrix mapping decoder hidden states to vocabulary logits
• $\text{softmax}(\cdot)$: the softmax function normalizing logits into a probability distribution over the vocabulary $V$

Training uses teacher forcing: the model is trained to predict each ground-truth token given the true previous tokens, using cross-entropy loss summed over all positions:

$$\mathcal{L} = -\sum_{t=1}^{T} \log P(y_t^* \mid y_1^*, \ldots, y_{t-1}^*, x_1, \ldots, x_n)$$

where:

• $y_t^*$: the ground-truth (reference) token at position $t$
• $T$: the total length of the target (reference) summary
• The summation accumulates cross-entropy loss across all target positions

Teacher forcing introduces a subtle training-inference mismatch: during training, the model always sees the correct previous token, but during inference, it sees its own (possibly wrong) previous predictions. This "exposure bias" can cause errors to compound during inference. Scheduled sampling and other techniques partially address this, but it remains a known limitation of the standard training procedure. In practice, the quality of large pretrained models has reduced the practical significance of exposure bias compared to earlier, smaller models.

Vanilla seq2seq models compress the entire source document into a fixed-length vector before decoding. For long documents, this creates a severe bottleneck: the model must memorize everything it might need from thousands of tokens into a single vector. Empirically, the performance of vanilla RNN-based seq2seq models degrades sharply as source length increases beyond a few dozen tokens. The information content of a thousand-word document simply cannot be faithfully compressed into a vector of a few hundred dimensions.

The attention mechanism, which we introduced in Chapter 10, solves this problem by allowing the decoder to selectively attend to relevant parts of the source at each generation step. At decoder step $t$, attention computes a context vector $c_t$ as a weighted sum of encoder hidden states:

$$c_t = \sum_{i=1}^{n} \alpha_{ti} \cdot h_i$$

where the attention weight $\alpha_{ti}$ measures how relevant source token $i$ is when generating summary token $t$. The weights are computed using a softmax over unnormalized alignment scores $e_{ti}$:

$$\alpha_{ti} = \frac{\exp(e_{ti})}{\sum_{j=1}^{n} \exp(e_{tj})}$$

The alignment score $e_{ti}$ is computed using a small learned network that receives the current decoder state and each encoder hidden state:

$$e_{ti} = v_a^\top \tanh(W_a s_{t-1} + U_a h_i)$$

where:

• $s_{t-1}$: the decoder hidden state at the previous time step $t-1$
• $h_i$: the encoder hidden state for source token $i$
• $W_a \in \mathbb{R}^{d \times d}$: a learned weight matrix applied to the decoder state
• $U_a \in \mathbb{R}^{d \times d}$: a learned weight matrix applied to the encoder state
• $v_a \in \mathbb{R}^d$: a learned vector projecting the combined representation to a scalar score

The context vector $c_t$ is then concatenated with the decoder state $s_t$ to produce the final representation used for predicting $y_t$. This allows the model to "look back" at the source with different weights at each generation step, focusing on the relevant parts of the document for each summary word.

For a document about a scientific study, when generating the word "participants," the attention mechanism would naturally weight encoder states corresponding to source tokens like "subjects," "volunteers," or "patients" more heavily than tokens describing the methodology. This selective reading is necessary for summarization: different parts of the source are relevant at different points in the generation. The attention weights effectively implement a soft alignment between source positions and generation steps.

Attention also provides a useful interpretability tool. By visualizing the attention weights $\alpha_{ti}$ as a matrix with summary positions on one axis and source positions on the other, you can see which source passages the model "read" while generating each summary word. This is far more interpretable than the black-box vector encoding of vanilla seq2seq, and it allows practitioners to diagnose cases where the model attended to the wrong source passages.

A critical limitation of vanilla seq2seq models for summarization is their inability to reproduce uncommon words, names, or technical terms from the source. If "CRISPR-Cas9" appears in the source document but not in the training vocabulary, the decoder cannot generate it. It might replace it with "the gene editing technique" or, worse, hallucinate an incorrect term entirely.

This problem is not rare. News articles are dense with proper names, place names, and numerical figures. Scientific papers use highly technical vocabulary. Legal documents reference specific statutes, parties, and clause numbers. Medical records contain drug names, dosages, and diagnostic codes. In all these domains, faithfully reproducing terms from the source is essential, yet a fixed vocabulary decoder cannot guarantee it. A model that paraphrases every proper noun it encounters will produce fluent but potentially misleading summaries.

The copy mechanism, introduced by See et al. (2017) in their Pointer-Generator Networks, addresses this by allowing the decoder to either generate a word from the vocabulary or copy a word directly from the source. A soft switch $p_{\text{gen}} \in [0,1]$ balances these options:

$$P(w) = p_{\text{gen}} \cdot P_{\text{vocab}}(w) + (1 - p_{\text{gen}}) \cdot \sum_{i:\, x_i = w} \alpha_{ti}$$

where:

• $p_{\text{gen}}$: the probability of generating from vocabulary, computed from the current context as $\sigma(w_h^\top h^* + w_s^\top s_t + w_x^\top x_t + b_{\text{gen}})$
• $P_{\text{vocab}}(w)$: the softmax distribution over the fixed vocabulary from the decoder
• $\sum_{i:\, x_i = w} \alpha_{ti}$: the total attention weight across all source positions $i$ where the token equals $w$, representing the copy probability for word $w$

If $p_{\text{gen}}$ is low, the model largely copies the most attended source tokens. If it is high, the model generates from its learned vocabulary. The model learns to use this switch contextually: proper nouns and rare terms get low $p_{\text{gen}}$ while common words and paraphrases get high $p_{\text{gen}}$.

The copy mechanism also provides a natural way to handle out-of-vocabulary tokens. Even if a word was never seen during training, the model can copy it from the source whenever the attention weights concentrate on the corresponding source positions. This is particularly important for numerical values, which would otherwise need to be reproduced character by character or memorized during training. A study in a clinical setting might mention a specific patient age (73 years), a specific blood pressure reading (142/91), and a specific medication dose (10 mg twice daily). A copy-enabled model can reproduce all three values accurately; a vocabulary-only model might substitute plausible but incorrect numbers.

Even with a copy mechanism, early attention-based summarizers suffered from a specific failure mode: they would repeat phrases from the source multiple times in the same summary. The model might generate "the Amazon rainforest" three times in a three-sentence summary because the most prominent source tokens kept attracting high attention weights.

See et al. (2017) also introduced a coverage mechanism to address this. A coverage vector $c_t$ accumulates the attention weights from all previous decoder steps:

$$c_t = \sum_{t'=0}^{t-1} \alpha_{t'}$$

This vector tracks which source tokens have already been attended to. A coverage loss then penalizes the model for attending to source positions that have already been heavily covered:

$$\text{cov\_loss}_t = \sum_i \min(\alpha_{ti}, c_{ti})$$

The $\min$ function captures the amount by which current attention overlaps with past attention. If token $i$ has already accumulated coverage $c_{ti} = 0.8$ and receives attention $\alpha_{ti} = 0.5$ at step $t$, the coverage loss contribution is $\min(0.5, 0.8) = 0.5$, penalizing this overlap. The total coverage loss is added to the main cross-entropy loss during training, pushing the model to spread its attention more evenly across source positions rather than repeatedly attending to the same ones.

Coverage mechanisms address repetition at the generation level, but they do not fully solve the deeper problem of hallucination. A model can generate novel hallucinated content even when it is not repeating itself. The distinction between repetition and hallucination is important for diagnosis: repetition means attending to the same correct source content too many times; hallucination means generating content with no correspondence to any source content.

Modern abstractive summarization is dominated by large pretrained Transformer models, building on the architectures we studied in Chapters 12 through 19. Models like BART and T5 are trained with pretraining objectives specifically designed for text-to-text generation, then fine-tuned on summarization datasets.

BART (Lewis et al., 2020) uses a denoising objective: the model is trained to reconstruct original text from various corrupted versions, including sentence permutation, token deletion, and text infilling. This pretraining teaches the model to understand text structure at a deep level, making it particularly effective for summarization. The key insight is that a model that can reconstruct a document from a heavily corrupted version must have developed a strong understanding of what makes a document coherent and informative. When fine-tuned on CNN/DailyMail or XSum, BART produces fluent, accurate summaries that substantially outperform earlier seq2seq approaches.

T5 (Raffel et al., 2020) frames all NLP tasks as text-to-text problems. Summarization is simply a task where the input is "summarize: [document]" and the output is the summary. This unified framing allows T5 to transfer knowledge across tasks: understanding developed while learning translation helps with summarization, and vice versa. Its large-scale pretraining on the Colossal Clean Crawled Corpus (C4) gives it broad language knowledge that transfers well to specialized summarization domains.

More recently, instruction-tuned large language models like GPT-4, Claude, and Llama have become the de facto choice for summarization in production applications. These models can follow detailed instructions: "Summarize in three bullet points, stressing the financial implications" or "Write a one-paragraph summary suitable for a general audience." This instruction-following capability, which earlier seq2seq models lacked, substantially increases the practical value of abstractive summarization. You are no longer limited to a fixed output format trained during fine-tuning; you can specify exactly what you want and the model adapts.

The key advantage of pretrained Transformers over earlier seq2seq models comes from scale and from the quality of the learned representations. A BART encoder that has processed hundreds of billions of words has learned rich semantic and syntactic structure that allows it to identify the key information in a document far more reliably than a model trained only on a summarization dataset. Pretraining provides the general language understanding; fine-tuning on summarization data teaches the model the specific output format and style. This division of labor between pretraining and fine-tuning is a pattern we have seen throughout this handbook, and it is particularly consequential for summarization because the task requires such broad language understanding.