Copy Mechanism › Copy Mechanism for Summarization
Language AI HandbookAll chapters Copy Mechanism › Copy Mechanism for SummarizationOne section · Sections in this chapter7 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Abstractive summarization is the canonical application for copy mechanisms. Summaries must preserve key facts, names, and numbers from the source document while also rephrasing and condensing the content. The pointer-generator architecture excels at this balance.
The tension between copying and abstracting is fundamental to what makes summarization hard. A system that only copies produces extractive summaries: verbatim passages from the source document, perhaps assembled in a different order. Extractive systems are reliable but inflexible; they cannot produce concise phrases that condense multiple source sentences, and they often produce stilted text when copied fragments are placed in new contexts. Abstractive systems that only generate may produce fluent, natural-sounding text, but they are prone to substituting wrong names, inventing statistics, or missing critical details. The pointer-generator sits at the ideal point in this spectrum: it generates fluent paraphrases for the parts of the summary that can be paraphrased, and copies verbatim for the parts that must be preserved exactly.
News summarization makes the stakes concrete. If a model summarizes "The Federal Reserve raised interest rates by 50 basis points" as "The Federal Reserve changed interest rates," the quantitative detail is lost. If it says "The Federal Reserve raised rates by 25 basis points," a factual error has been introduced. A pointer-generator trained on financial news learns that numerical values like "50 basis points" appear frequently enough in source documents but rarely in summaries as generated text, and it routes them through the copy channel. The result is higher ROUGE scores and more factually reliable summaries.
# Simulate summarization scenario
source_document = """
Dr. Sarah Chen, a researcher at Stanford University, announced a breakthrough
in quantum computing. The discovery, published in Nature on March 15, could
reduce error rates by 47 percent. Chen's team worked with IBM and Google
on the three-year project.
"""
# Words that MUST be copied (rare/specific)
must_copy = [
"Sarah",
"Chen",
"Stanford",
"Nature",
"March",
"15",
"47",
"IBM",
"Google",
]
# Words that could be generated (common)
can_generate = [
"researcher",
"announced",
"breakthrough",
"discovery",
"published",
"reduce",
"team",
"project",
]
# Build a simple vocabulary (missing the rare words)
simple_vocab = [
"<pad>",
"<unk>",
"<sos>",
"<eos>",
"the",
"a",
"in",
"at",
"on",
"by",
"and",
"to",
"of",
"that",
"which",
"percent",
"researcher",
"announced",
"breakthrough",
"discovery",
"published",
"reduce",
"team",
"project",
"university",
"computing",
"quantum",
"error",
"rates",
"year",
"three",
]Summarization Vocabulary Analysis ================================================== Source document excerpt: ' Dr. Sarah Chen, a researcher at Stanford University, announced a breakthrough in quantum computing...' Critical words that must be copied: Sarah: ✗ OOV - needs copy Chen: ✗ OOV - needs copy Stanford: ✗ OOV - needs copy Nature: ✗ OOV - needs copy March: ✗ OOV - needs copy 15: ✗ OOV - needs copy 47: ✗ OOV - needs copy IBM: ✗ OOV - needs copy Google: ✗ OOV - needs copy Common words that can be generated: researcher: ✓ in vocab announced: ✓ in vocab breakthrough: ✓ in vocab discovery: ✓ in vocab
The vocabulary analysis reveals the structural challenge. Out of nine critical words in the source document, the simple vocabulary covers none of them. Every important factual detail, the researcher's name, the institution, the journal, the date, the percentage, and the partner companies, falls outside the vocabulary. A standard seq2seq decoder would replace all of them with <unk> tokens, producing a summary that is grammatically correct but conveys almost no useful information. With copying enabled, each of these words can be reproduced directly from the source.
Let's trace through how the model would generate a summary:
# Simulated generation trace
generation_trace = [
{"token": "Dr.", "source": "copy", "p_gen": 0.15, "attn_peak": "Dr."},
{"token": "Chen", "source": "copy", "p_gen": 0.08, "attn_peak": "Chen"},
{
"token": "announced",
"source": "generate",
"p_gen": 0.82,
"attn_peak": None,
},
{"token": "a", "source": "generate", "p_gen": 0.91, "attn_peak": None},
{
"token": "quantum",
"source": "generate",
"p_gen": 0.73,
"attn_peak": None,
},
{
"token": "computing",
"source": "generate",
"p_gen": 0.78,
"attn_peak": None,
},
{
"token": "breakthrough",
"source": "generate",
"p_gen": 0.85,
"attn_peak": None,
},
{"token": "at", "source": "generate", "p_gen": 0.88, "attn_peak": None},
{
"token": "Stanford",
"source": "copy",
"p_gen": 0.12,
"attn_peak": "Stanford",
},
{"token": ".", "source": "generate", "p_gen": 0.95, "attn_peak": None},
]Generation Trace ============================================================ Generating summary: 'Dr. Chen announced a quantum computing breakthrough at Stanford.' Token Source p_gen Attention Peak ------------------------------------------------------- Dr. copy 0.15 Dr. Chen copy 0.08 Chen announced generate 0.82 - a generate 0.91 - quantum generate 0.73 - computing generate 0.78 - breakthrough generate 0.85 - at generate 0.88 - Stanford copy 0.12 Stanford . generate 0.95 - Note: Low p_gen indicates copying; high p_gen indicates generation
