Part of Language AI Handbook
Explains how copy mechanisms enable seq2seq models to handle out-of-vocabulary words.
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Copy Mechanism
Standard sequence-to-sequence models generate output tokens by selecting from a fixed vocabulary. This works well for common words, but what happens when the input contains a rare proper name, a technical term, or a number that the model has never seen? The decoder must either hallucinate a substitute or produce a generic placeholder. Neither outcome is satisfactory.
Copy mechanisms solve this problem by allowing the decoder to directly copy tokens from the input sequence. Instead of being forced to generate every output token from scratch, the model can "point" to input positions and reproduce those tokens verbatim. This matters for tasks like summarization, where preserving names and facts is critical, and for handling out-of-vocabulary words that would otherwise be lost.
Think of it like the difference between a journalist writing from memory versus one with the original press release in hand. The journalist writing from memory must rely on their prior knowledge, which may not include the exact spelling of a foreign diplomat's name, a precise dollar figure, or a specific product model number. The journalist with the source document in front of them can simply copy those details verbatim while still composing their own narrative around them. Copy mechanisms give neural models that same advantage: access to the source text as a direct lookup resource, not just as a context to be encoded and compressed into a fixed hidden state.
The challenge is architectural. A standard decoder outputs a probability distribution over a fixed vocabulary at each step. Names like "Nakamura" or "Pfizer" or numbers like "47.3%" may simply not exist in that vocabulary, meaning no matter how much the model attends to them in the encoder, it has no mechanism to reproduce them. The copy mechanism threads a new pathway from the attention distribution directly to the output, bypassing the vocabulary bottleneck for tokens that are better reproduced than generated.
This chapter explores how copy mechanisms work, from the foundational pointer networks to the practical pointer-generator architecture. You'll learn how models compute copy probabilities, blend copying with generation, and handle the challenging case of out-of-vocabulary tokens. By the end, you'll understand why copy mechanisms became a standard component in production summarization systems and how they relate to the broader challenge of faithfulness in neural text generation.
The Vocabulary Problem in Generation
Before diving into copy mechanisms, let's understand the problem they solve. Traditional seq2seq models have a fixed output vocabulary, typically the most frequent words in the training data. This design choice is almost inevitable: the final softmax layer must produce a probability over some finite set of tokens, and the practical size of that set is constrained by memory and the need for sufficient training examples per token. A vocabulary of 50,000 words covers the vast majority of everyday language, but it covers only a small fraction of all the proper nouns, technical terms, product names, and numerical values that appear in real-world text.
The problem is asymmetric: a model trained on a large corpus will have no difficulty generating words like "the," "announced," or "discovery," but it will fail entirely on words like "Nakamura," "CRISPR," or "Q3-FY2024." These words may appear once or twice in the training data, not enough to learn a reliable embedding, or they may never appear at all if they postdate the training cutoff. When the model encounters such words at inference time, it has no representation for them. The only option the standard vocabulary distribution offers is the catch-all <unk> token, which carries no information about the actual word that should be there.
from collections import Counter
# Simulate a typical vocabulary scenario
training_corpus = """
The president announced new policies today.
Scientists discovered a breakthrough in medicine.
The company reported strong quarterly earnings.
Researchers published findings in Nature journal.
The government proposed infrastructure spending.
"""
# Build vocabulary from training data
words = training_corpus.lower().split()
word_counts = Counter(words)
# Typical vocab: keep only frequent words
vocab_size = 20
vocab = ["<unk>", "<sos>", "<eos>"] + [
word for word, _ in word_counts.most_common(vocab_size - 3)
]
word_to_idx = {word: i for i, word in enumerate(vocab)}Vocabulary Construction ================================================== Vocabulary size: 20 Top words in vocab: <unk> <sos> <eos> the in president announced new policies today. ...
Now consider what happens when we encounter input with rare words:
# Input with rare/unseen words
test_input = "Dr. Nakamura presented findings at the Stanford conference."
# Check which words are in vocabulary
in_vocab = []
out_of_vocab = []
for word in test_input.lower().replace(".", "").split():
if word in word_to_idx:
in_vocab.append(word)
else:
out_of_vocab.append(word)Out-of-Vocabulary Analysis ================================================== Input: Dr. Nakamura presented findings at the Stanford conference. In vocabulary: ['the'] Out of vocabulary (OOV): ['dr', 'nakamura', 'presented', 'findings', 'at', 'stanford', 'conference'] OOV rate: 87.5%
The words "Nakamura," "Stanford," and "conference" are critical for an accurate summary, yet a standard decoder cannot produce them. It would have to replace them with <unk> tokens or substitute similar but incorrect words. Copy mechanisms solve this directly: let the decoder point to these words in the input and copy them.
This capability directly affects factual accuracy. For tasks like question answering, summarization, and dialogue, names and numbers must be reproduced exactly. Replacing "Stanford" with "the university" or "Nakamura" with a <unk> token defeats the system.s purpose. The OOV problem is especially severe for named entities, which are precisely the words that carry the most information and are the most difficult to recover from context alone.
Pointer Networks: The Foundation
To understand copy mechanisms, we must first grapple with a fundamental question: how can a neural network select items from a variable-length input sequence? Traditional neural networks produce outputs of fixed size, but when copying from input, we need to point to one of positions where changes with each input.
Pointer networks, introduced by Vinyals et al. in 2015, solve this by repurposing attention. Recall from earlier chapters that attention computes a weighted combination of encoder states, producing weights that sum to 1 across all input positions. These weights already form a probability distribution over the input. The key insight is simple: instead of using attention weights only to compute context vectors, we can interpret them directly as "pointing" probabilities.
Pointer networks (Vinyals et al., 2015) were originally developed not for text generation but for combinatorial optimization problems like the Traveling Salesman Problem, where the output must be a permutation of the input. The model needed to output sequences like "visit city 3, then city 1, then city 5," selecting from a set of cities whose size varied between problem instances. Standard softmax over a fixed vocabulary could not handle this because the set of cities changes with each input. The pointer network's solution, treating attention weights as selection probabilities, turned out to be equally applicable to text generation, where the "cities" are input tokens.
The original motivation from combinatorial optimization is instructive. In the TSP setting, the vocabulary is literally the input itself: the model must output pointers into the same sequence it reads. This framing makes it obvious that attention weights are the right abstraction, because attention already computes a relevance score for each input position. In text generation, the insight carries over: the source tokens form a dynamic vocabulary, and the attention distribution is already computing something very close to "how relevant is each source token right now?" Pointer networks simply formalize this connection and use it directly as a selection mechanism.
What makes pointer networks elegant is that they add almost nothing to a standard attention-based seq2seq model. The attention mechanism was already there; the pointer network just changes how you interpret its output. Instead of multiplying attention weights by encoder states to get a context vector, you use those same weights directly as a probability distribution over which input position to point to. The architectural change is minimal, but the capability it adds is substantial.
A pointer network is a sequence-to-sequence model where the output at each step is a pointer to an element in the input sequence. Instead of generating tokens from a vocabulary, it uses attention over the input to select which input position to "point to."
Let's trace through the mechanics. At each decoder step, we have a decoder hidden state that encodes what we've generated so far. We also have encoder outputs representing each input position. The pointer mechanism computes a score for each input position, measuring how relevant that position is given the current decoder state:
- Project both representations into a common space using learned weight matrices
- Combine them through addition (Bahdanau-style) and pass through a nonlinearity
- Compute a scalar score for each position using a learned vector
- Normalize with softmax to obtain a valid probability distribution
The resulting probabilities tell us: "Given what I've generated so far, how likely should I point to each input position?"
In practice, the scoring function is the same Bahdanau additive attention you encountered in the attention chapter. The difference is purely in how the output is used. In standard attention, the weights are intermediate values used to compute a context vector. In a pointer network, the weights are the final output: they are the model's answer to the question "which input token should I select?" This reuse of existing machinery is why pointer networks can be integrated into seq2seq models with minimal architectural overhead.
import torch
import torch.nn as nn
import torch.nn.functional as F
class PointerAttention(nn.Module):
"""
Attention mechanism that produces pointer probabilities.
"""
def __init__(self, hidden_dim):
super().__init__()
self.hidden_dim = hidden_dim
# Attention parameters (Bahdanau-style)
self.W_encoder = nn.Linear(hidden_dim, hidden_dim, bias=False)
self.W_decoder = nn.Linear(hidden_dim, hidden_dim, bias=False)
self.v = nn.Linear(hidden_dim, 1, bias=False)
def forward(self, decoder_state, encoder_outputs, mask=None):
"""
Compute pointer probabilities over input positions.
Args:
decoder_state: (batch, hidden_dim)
encoder_outputs: (batch, seq_len, hidden_dim)
mask: (batch, seq_len) - True for valid positions
Returns:
pointer_probs: (batch, seq_len) - probability of pointing to each input
"""
batch_size, seq_len, _ = encoder_outputs.shape
# Project encoder outputs: (batch, seq_len, hidden)
encoder_proj = self.W_encoder(encoder_outputs)
# Project decoder state: (batch, hidden) -> (batch, 1, hidden)
decoder_proj = self.W_decoder(decoder_state).unsqueeze(1)
# Compute attention scores: (batch, seq_len, 1) -> (batch, seq_len)
scores = self.v(torch.tanh(encoder_proj + decoder_proj)).squeeze(-1)
# Apply mask (set invalid positions to -inf)
if mask is not None:
scores = scores.masked_fill(~mask, float("-inf"))
# Softmax to get pointer probabilities
pointer_probs = F.softmax(scores, dim=-1)
return pointer_probs
# Create example
hidden_dim = 64
batch_size = 2
seq_len = 5
attention = PointerAttention(hidden_dim)
decoder_state = torch.randn(batch_size, hidden_dim)
encoder_outputs = torch.randn(batch_size, seq_len, hidden_dim)
mask = torch.ones(batch_size, seq_len, dtype=torch.bool)
pointer_probs = attention(decoder_state, encoder_outputs, mask)Pointer Attention Output ================================================== Input sequence length: 5 Pointer probabilities (batch 0): Position 0: 0.161 ████ Position 1: 0.243 ███████ Position 2: 0.214 ██████ Position 3: 0.163 ████ Position 4: 0.218 ██████ Sum of probabilities: 1.0000
The pointer probabilities form a valid distribution over input positions. At each decoding step, the model can select which input token to copy by sampling from or taking the argmax of this distribution.
Notice that the sum of pointer probabilities equals exactly 1.0. This is a direct consequence of the softmax normalization, and it is an important property: it means we can interpret these values as valid probabilities and mix them with other probability distributions in a mathematically principled way, which is exactly what the pointer-generator network will do in the next section.
Computing the Copy Probability
A pure pointer network can only copy from the input, but real text generation requires both copying and generating. Consider summarizing "Dr. Chen announced the results." We want to copy "Dr. Chen" (a rare name) but generate common words like "announced" and "the" from our vocabulary. How does the model decide which strategy to use at each step?
The solution introduces a soft switch between two modes: generating from vocabulary versus copying from input. Rather than making a hard binary choice, we compute a generation probability that smoothly interpolates between the two strategies. When is high, the model favors generation; when low, it favors copying.
The key insight is that a hard binary choice would be problematic for several reasons. First, the correct strategy is not always obvious from the current word alone. For a word like "bank," whether to copy it from the source or generate it from vocabulary depends on whether the source document is about finance or rivers, which is a semantic question that requires context. Second, hard decisions create discontinuities in the computation graph that make gradient-based training difficult. The soft switch produces smooth gradients throughout, allowing the model to learn the copy-versus-generate decision jointly with all other parameters.
What information should determine this switch? Intuitively, the decision depends on:
- The context vector : What part of the input is the model attending to? If attention focuses on a rare word, copying makes sense.
- The decoder state : What has the model generated so far? This captures the "momentum" of the generation process.
- The previous token : What did we just output? After generating "Dr.", we likely want to copy the following name.
We combine these three signals through a linear combination, then squash the result to using the sigmoid function:
where:
- : the probability of generating from the vocabulary (vs. copying from input)
- : the sigmoid function, , which maps any real number to the range
- : the context vector from attention at decoder step
- : the decoder hidden state at step
- : the decoder input embedding at step (typically the previous output token)
- : learnable weight vectors that determine how much each input contributes to the decision
- : a learnable bias term that shifts the default behavior toward generating or copying
The model learns the weights during training. It discovers patterns like "when the context vector indicates a rare word and the decoder state suggests we're in the middle of a named entity, favor copying." These patterns emerge from the data without explicit programming.
The choice of three input signals reflects thoughtful design. The context vector tells the model what part of the source it is currently attending to: if attention is concentrated on a rare proper noun, copying is likely appropriate. The decoder state captures the sequential context of what has been generated: after producing "Dr." the model should understand it is likely in a name sequence and prepare to copy the following word. The previous token embedding provides the most immediate signal: the specific word just generated often strongly implies what comes next. Together these three sources of information give the switch enough context to make good decisions consistently.
class CopySwitch(nn.Module):
"""
Computes the probability of generating vs copying.
"""
def __init__(self, hidden_dim, embed_dim):
super().__init__()
# Linear combination for p_gen
self.w_context = nn.Linear(hidden_dim, 1, bias=False)
self.w_state = nn.Linear(hidden_dim, 1, bias=False)
self.w_input = nn.Linear(embed_dim, 1, bias=False)
self.bias = nn.Parameter(torch.zeros(1))
def forward(self, context, decoder_state, decoder_input):
"""
Compute generation probability.
Args:
context: (batch, hidden_dim) - attention context vector
decoder_state: (batch, hidden_dim) - decoder hidden state
decoder_input: (batch, embed_dim) - current input embedding
Returns:
p_gen: (batch, 1) - probability of generating from vocabulary
"""
score = (
self.w_context(context)
+ self.w_state(decoder_state)
+ self.w_input(decoder_input)
+ self.bias
)
p_gen = torch.sigmoid(score)
return p_gen
# Example
embed_dim = 32
copy_switch = CopySwitch(hidden_dim, embed_dim)
context = torch.randn(batch_size, hidden_dim)
decoder_input = torch.randn(batch_size, embed_dim)
p_gen = copy_switch(context, decoder_state, decoder_input)Copy Switch Output
==================================================
Generation probability p_gen:
Batch 0: p_gen = 0.512
p_copy = 0.488
Batch 1: p_gen = 0.293
p_copy = 0.707The output reveals how the copy switch behaves: values near 0 indicate the model prefers copying, while values near 1 indicate generation. In practice, the model learns to produce low for rare words and names, and high for common function words.
Empirically, well-trained pointer-generator models develop strong separation in values across token types. Articles, prepositions, and common verbs like "the," "of," and "said" typically receive above 0.8. Named entities and numbers typically receive below 0.2. The interesting cases are mid-frequency words that could reasonably come from either source, where the model must use context to decide. This learned specialization is not hand-coded: it emerges entirely from exposure to (source, summary) pairs during training.
Mixing Generation and Copying
We now have two probability distributions: one over the vocabulary (from the decoder's softmax) and one over input positions (from pointer attention). We also have a switch that tells us how much to trust each. How do we combine them into a single output distribution?
The naive approach would be to make a hard choice: either generate or copy. But this loses information and creates discontinuities that hurt gradient flow during training. Instead, we use as a mixing coefficient that smoothly blends both distributions.
Think of the mixing operation as a weighted average of two experts: the generator expert, which knows the full vocabulary and can produce fluent language, and the copier expert, which has direct access to the source tokens regardless of whether they appear in any vocabulary. The parameter controls how much weight to give each expert. Rather than forcing a discrete commitment, the model holds both possibilities open simultaneously, which is both more numerically stable during training and more flexible at inference.
Consider a word that we might want to output. There are three cases:
-
is only in the vocabulary: It can only be generated, so its probability comes entirely from , scaled by .
-
is only in the input (OOV): It can only be copied, so its probability comes from the attention weights on positions containing , scaled by .
-
is in both: It receives probability from both sources, which are added together.
This leads to the final probability formula:
where:
- : the final probability of outputting word
- : the generation probability from the copy switch
- : the probability assigned to by the decoder's vocabulary softmax
- : the attention weight (pointer probability) for input position
- : the token at input position
- : sum over all positions where the input token equals
The summation deserves attention. If a word appears multiple times in the input, for example "the" might appear at positions 2, 7, and 15, we sum the attention weights from all those positions. This makes intuitive sense: if the model attends to any occurrence of "the" in the input, that attention contributes to the probability of outputting "the."
This summation over all matching positions is a subtle but important design choice. An alternative would be to route attention from each position independently, but that would require knowing in advance which specific occurrence of a repeated word you want to copy. By aggregating over all positions containing the same word, the model sidesteps this ambiguity: it only needs to decide what word to output, not which occurrence to point to.
Let's trace through a concrete example. Suppose we're generating a summary and the input contains "the president Nakamura said." Our vocabulary includes common words but not "Nakamura." At the current step:
- The vocabulary distribution assigns: ,
- The pointer distribution assigns: (the), (president), (Nakamura), (said)
- The copy switch outputs: (favoring copying)
For "Nakamura" (OOV, only copyable):
For "said" (in both vocab and input):
The OOV word "Nakamura" receives substantial probability through copying alone, while "said" gets a boost from both sources.
This worked example illustrates a key structural property of the mechanism: words that exist only in the vocabulary rely entirely on , words that exist only in the source rely entirely on , and words that exist in both receive contributions from both channels. The total probability remains valid (sums to 1) because both component distributions are individually normalized and the mixing coefficients and sum to 1. This mathematical cleanliness is one of the reasons the pointer-generator formulation became the standard approach.
def compute_final_distribution(
p_gen, # (batch, 1) generation probability
vocab_dist, # (batch, vocab_size) vocabulary distribution
pointer_probs, # (batch, src_len) attention/pointer distribution
source_ids, # (batch, src_len) token IDs in source
vocab_size, # int
oov_ids=None, # (batch, src_len) IDs for OOV tokens (optional)
):
"""
Combine generation and copy distributions.
For words in both vocab and source, probabilities are summed.
For OOV words (only in source), only copy probability applies.
"""
batch_size, src_len = source_ids.shape
# Start with generation distribution, scaled by p_gen
# Extended vocab includes OOV slots
if oov_ids is not None:
max_oov = oov_ids.max().item() + 1
extended_size = vocab_size + max_oov
else:
extended_size = vocab_size
final_dist = torch.zeros(batch_size, extended_size)
final_dist[:, :vocab_size] = p_gen * vocab_dist
# Add copy probabilities
p_copy = 1 - p_gen # (batch, 1)
for b in range(batch_size):
for i in range(src_len):
token_id = source_ids[b, i].item()
# If token is in vocab, add to that position
if token_id < vocab_size:
final_dist[b, token_id] += p_copy[b, 0] * pointer_probs[b, i]
# If OOV, add to extended vocab position
elif oov_ids is not None:
oov_idx = oov_ids[b, i].item()
final_dist[b, vocab_size + oov_idx] += (
p_copy[b, 0] * pointer_probs[b, i]
)
return final_dist
# Example with a small vocabulary
small_vocab = ["<unk>", "<sos>", "<eos>", "the", "said", "president"]
small_vocab_size = len(small_vocab)
# Source tokens: "the president Nakamura said"
# "Nakamura" is OOV (id = vocab_size, oov_idx = 0)
source_ids = torch.tensor([[3, 5, 6, 4]]) # the, president, <oov>, said
oov_ids = torch.tensor([[0, 0, 0, 0]]) # Only position 2 is OOV
# Simulated distributions
vocab_dist = F.softmax(torch.randn(1, small_vocab_size), dim=-1)
pointer_probs_example = torch.tensor(
[[0.1, 0.3, 0.4, 0.2]]
) # High attention on "Nakamura"
p_gen_example = torch.tensor([[0.3]]) # Likely to copy
final_dist = compute_final_distribution(
p_gen_example,
vocab_dist,
pointer_probs_example,
source_ids,
small_vocab_size,
oov_ids,
)Final Distribution Computation ================================================== Source: ['the', 'president', 'Nakamura', 'said'] p_gen = 0.30 (low = prefer copying) Pointer probabilities: the: 0.10 president: 0.30 Nakamura: 0.40 said: 0.20 Final distribution: <unk>: 0.023 <sos>: 0.020 <eos>: 0.148 ████ the: 0.111 ███ said: 0.178 █████ president: 0.240 ███████ Nakamura (OOV): 0.280 ████████
With $p_{\text{gen}} = 0.30$:
| Word | In Vocab | In Source | Generation | Copy | **Total** |
|:-----|:--------:|:---------:|-----------:|-----:|----------:|
| the | ✓ | ✓ | 0.041 | 0.070 | **0.111** |
| said | ✓ | ✓ | 0.038 | 0.140 | **0.178** |
| president | ✓ | ✓ | 0.030 | 0.210 | **0.240** |
| Nakamura (OOV) | . | ✓ | 0.000 | 0.280 | **0.280** |: Final probability distribution combining generation and copy contributions. Words in both vocabulary and source (like "the" and "said") receive probability from both channels. OOV words like "Nakamura" can only be copied, receiving their entire probability from the copy mechanism. {#tbl-probability-combination}
The table reveals the mechanics of probability combination. "Nakamura" receives its entire probability from copying, since it's not in the vocabulary. "the" and "said" appear in both the source and vocabulary, so they receive contributions from both channels. With , the copy channel dominates, which is appropriate when the model needs to preserve specific names from the input.
The output confirms our mathematical analysis. "Nakamura," despite being out-of-vocabulary, receives the highest probability through the copy mechanism. The model can produce this word in its output even though it never appeared in training. Meanwhile, words like "said" and "the" receive probability from both generation and copying, with their contributions weighted by .
This combination of generation and copying solves the OOV problem while preserving the model's ability to generate fluent, grammatical text. The soft switch learns to route rare words through copying and common words through generation.
It is worth pausing to appreciate what this achieves at training time. The model never receives explicit supervision about which tokens should be copied and which should be generated. It only receives the final target sequence. Yet through backpropagation, it learns to route the gradient signal appropriately: when an OOV target token has high probability only through the copy channel, the gradient flows through the attention weights, encouraging the model to attend more sharply to the relevant source position. When a common token is generated correctly, the gradient flows through the vocabulary softmax. The two pathways train themselves to specialize without any manual annotation of copy versus generate decisions.
The Pointer-Generator Network
The pointer-generator network, introduced by See et al. (2017) for abstractive summarization, combines the ideas above into a cohesive architecture. It extends the standard attention-based seq2seq model with a copy mechanism, enabling it to both generate words from the vocabulary and copy words from the source document.
The 2017 paper by Abigail See, Peter Liu, and Christopher Manning was a landmark in neural summarization research. Their model was evaluated on the CNN/DailyMail dataset, a standard benchmark where news articles are summarized using bullet-point highlights. The pointer-generator model substantially outperformed prior neural approaches on ROUGE metrics, and more importantly, it produced summaries with dramatically fewer factual errors because it could copy names, numbers, and entities directly from the source. The paper also introduced the coverage mechanism discussed later in this chapter, addressing the separate problem of repetitive output. Together, these two contributions made the pointer-generator the go-to architecture for neural summarization for several years.
The architecture is elegant precisely because it adds so little to the baseline. The encoder is a standard bidirectional LSTM that processes the source document. The decoder is a standard LSTM with attention. The entire copy mechanism adds only two components: the copy switch (a small linear layer plus sigmoid) and the distribution combination step. The extended vocabulary and OOV tracking add some bookkeeping complexity, but no additional trainable parameters beyond those in the switch. This efficiency was a significant practical advantage when the alternative was large, complex architectures that were difficult to train.
Let's implement a complete pointer-generator decoder step:
class PointerGeneratorDecoder(nn.Module):
"""
Single decoding step of a pointer-generator network.
"""
def __init__(self, vocab_size, embed_dim, hidden_dim):
super().__init__()
self.vocab_size = vocab_size
self.hidden_dim = hidden_dim
# Embedding and LSTM
self.embedding = nn.Embedding(vocab_size, embed_dim)
self.lstm = nn.LSTMCell(embed_dim + hidden_dim, hidden_dim)
# Attention
self.attention = PointerAttention(hidden_dim)
# Copy switch
self.copy_switch = CopySwitch(hidden_dim, embed_dim)
# Vocabulary projection
self.vocab_proj = nn.Linear(hidden_dim * 2, vocab_size)
def forward(
self,
input_token,
prev_hidden,
prev_cell,
encoder_outputs,
source_ids,
encoder_mask=None,
):
"""
Perform one decoding step.
Args:
input_token: (batch,) - previous output token
prev_hidden: (batch, hidden) - previous hidden state
prev_cell: (batch, hidden) - previous cell state
encoder_outputs: (batch, src_len, hidden) - encoder outputs
source_ids: (batch, src_len) - source token IDs
encoder_mask: (batch, src_len) - valid source positions
Returns:
final_dist: (batch, extended_vocab) - output distribution
hidden: (batch, hidden) - new hidden state
cell: (batch, hidden) - new cell state
attn_weights: (batch, src_len) - attention weights
"""
batch_size = input_token.shape[0]
# Embed input
embedded = self.embedding(input_token) # (batch, embed)
# Compute attention over encoder outputs
attn_weights = self.attention(
prev_hidden, encoder_outputs, encoder_mask
)
# Context vector
context = torch.bmm(attn_weights.unsqueeze(1), encoder_outputs).squeeze(
1
)
# LSTM input: concatenate embedding and context
lstm_input = torch.cat([embedded, context], dim=-1)
# LSTM step
hidden, cell = self.lstm(lstm_input, (prev_hidden, prev_cell))
# Vocabulary distribution
vocab_input = torch.cat([hidden, context], dim=-1)
vocab_logits = self.vocab_proj(vocab_input)
vocab_dist = F.softmax(vocab_logits, dim=-1)
# Copy probability
p_gen = self.copy_switch(context, hidden, embedded)
# Combine distributions
final_dist = self._combine_distributions(
p_gen, vocab_dist, attn_weights, source_ids
)
return final_dist, hidden, cell, attn_weights
def _combine_distributions(
self, p_gen, vocab_dist, attn_weights, source_ids
):
"""Combine generation and copy distributions."""
batch_size, src_len = source_ids.shape
# For simplicity, assume no OOV (would need extended vocab otherwise)
final_dist = p_gen * vocab_dist
# Add copy probabilities
p_copy = 1 - p_gen
# Use scatter_add for efficiency
copy_dist = torch.zeros_like(vocab_dist)
copy_dist.scatter_add_(1, source_ids, attn_weights * p_copy)
final_dist = final_dist + copy_dist
return final_dist
# Test the decoder
vocab_size = 1000
embed_dim = 64
hidden_dim = 128
batch_size = 2
src_len = 10
decoder = PointerGeneratorDecoder(vocab_size, embed_dim, hidden_dim)
# Dummy inputs
input_token = torch.randint(0, vocab_size, (batch_size,))
prev_hidden = torch.zeros(batch_size, hidden_dim)
prev_cell = torch.zeros(batch_size, hidden_dim)
encoder_outputs = torch.randn(batch_size, src_len, hidden_dim)
source_ids = torch.randint(0, vocab_size, (batch_size, src_len))
final_dist, hidden, cell, attn = decoder(
input_token, prev_hidden, prev_cell, encoder_outputs, source_ids
)Pointer-Generator Decoder Output ================================================== Output distribution shape: torch.Size([2, 1000]) Hidden state shape: torch.Size([2, 128]) Attention weights shape: torch.Size([2, 10]) Top 5 predicted tokens (batch 0): Token 361: 0.0683 Token 101: 0.0669 Token 621: 0.0636 Token 86: 0.0592 Token 645: 0.0568 Distribution sum: 1.0000
The output distribution sums to 1.0, confirming it's a valid probability distribution. The decoder produces probabilities over all tokens in the vocabulary, and the top predictions show which tokens are most likely at this step. In practice, these probabilities would be used either for greedy decoding (selecting the argmax) or for beam search (maintaining multiple hypotheses).
The implementation above uses scatter_add_ for the copy distribution, which is an efficient tensor operation that accumulates copy probabilities into the right vocabulary positions in a single pass. This is important for performance: the naive loop over source positions that the compute_final_distribution function earlier used works for illustration purposes but would be far too slow for real-world documents with hundreds or thousands of tokens. The scatter operation achieves the same result in vectorized form, making it suitable for batched training.
Copy Mechanism for Summarization
Abstractive summarization is the canonical application for copy mechanisms. Summaries must preserve key facts, names, and numbers from the source document while also rephrasing and condensing the content. The pointer-generator architecture excels at this balance.
The tension between copying and abstracting is fundamental to what makes summarization hard. A system that only copies produces extractive summaries: verbatim passages from the source document, perhaps assembled in a different order. Extractive systems are reliable but inflexible; they cannot produce concise phrases that condense multiple source sentences, and they often produce stilted text when copied fragments are placed in new contexts. Abstractive systems that only generate may produce fluent, natural-sounding text, but they are prone to substituting wrong names, inventing statistics, or missing critical details. The pointer-generator sits at the ideal point in this spectrum: it generates fluent paraphrases for the parts of the summary that can be paraphrased, and copies verbatim for the parts that must be preserved exactly.
News summarization makes the stakes concrete. If a model summarizes "The Federal Reserve raised interest rates by 50 basis points" as "The Federal Reserve changed interest rates," the quantitative detail is lost. If it says "The Federal Reserve raised rates by 25 basis points," a factual error has been introduced. A pointer-generator trained on financial news learns that numerical values like "50 basis points" appear frequently enough in source documents but rarely in summaries as generated text, and it routes them through the copy channel. The result is higher ROUGE scores and more factually reliable summaries.
# Simulate summarization scenario
source_document = """
Dr. Sarah Chen, a researcher at Stanford University, announced a breakthrough
in quantum computing. The discovery, published in Nature on March 15, could
reduce error rates by 47 percent. Chen's team worked with IBM and Google
on the three-year project.
"""
# Words that MUST be copied (rare/specific)
must_copy = [
"Sarah",
"Chen",
"Stanford",
"Nature",
"March",
"15",
"47",
"IBM",
"Google",
]
# Words that could be generated (common)
can_generate = [
"researcher",
"announced",
"breakthrough",
"discovery",
"published",
"reduce",
"team",
"project",
]
# Build a simple vocabulary (missing the rare words)
simple_vocab = [
"<pad>",
"<unk>",
"<sos>",
"<eos>",
"the",
"a",
"in",
"at",
"on",
"by",
"and",
"to",
"of",
"that",
"which",
"percent",
"researcher",
"announced",
"breakthrough",
"discovery",
"published",
"reduce",
"team",
"project",
"university",
"computing",
"quantum",
"error",
"rates",
"year",
"three",
]Summarization Vocabulary Analysis ================================================== Source document excerpt: ' Dr. Sarah Chen, a researcher at Stanford University, announced a breakthrough in quantum computing...' Critical words that must be copied: Sarah: ✗ OOV - needs copy Chen: ✗ OOV - needs copy Stanford: ✗ OOV - needs copy Nature: ✗ OOV - needs copy March: ✗ OOV - needs copy 15: ✗ OOV - needs copy 47: ✗ OOV - needs copy IBM: ✗ OOV - needs copy Google: ✗ OOV - needs copy Common words that can be generated: researcher: ✓ in vocab announced: ✓ in vocab breakthrough: ✓ in vocab discovery: ✓ in vocab
The vocabulary analysis reveals the structural challenge. Out of nine critical words in the source document, the simple vocabulary covers none of them. Every important factual detail, the researcher's name, the institution, the journal, the date, the percentage, and the partner companies, falls outside the vocabulary. A standard seq2seq decoder would replace all of them with <unk> tokens, producing a summary that is grammatically correct but conveys almost no useful information. With copying enabled, each of these words can be reproduced directly from the source.
Let's trace through how the model would generate a summary:
# Simulated generation trace
generation_trace = [
{"token": "Dr.", "source": "copy", "p_gen": 0.15, "attn_peak": "Dr."},
{"token": "Chen", "source": "copy", "p_gen": 0.08, "attn_peak": "Chen"},
{
"token": "announced",
"source": "generate",
"p_gen": 0.82,
"attn_peak": None,
},
{"token": "a", "source": "generate", "p_gen": 0.91, "attn_peak": None},
{
"token": "quantum",
"source": "generate",
"p_gen": 0.73,
"attn_peak": None,
},
{
"token": "computing",
"source": "generate",
"p_gen": 0.78,
"attn_peak": None,
},
{
"token": "breakthrough",
"source": "generate",
"p_gen": 0.85,
"attn_peak": None,
},
{"token": "at", "source": "generate", "p_gen": 0.88, "attn_peak": None},
{
"token": "Stanford",
"source": "copy",
"p_gen": 0.12,
"attn_peak": "Stanford",
},
{"token": ".", "source": "generate", "p_gen": 0.95, "attn_peak": None},
]Generation Trace ============================================================ Generating summary: 'Dr. Chen announced a quantum computing breakthrough at Stanford.' Token Source p_gen Attention Peak ------------------------------------------------------- Dr. copy 0.15 Dr. Chen copy 0.08 Chen announced generate 0.82 - a generate 0.91 - quantum generate 0.73 - computing generate 0.78 - breakthrough generate 0.85 - at generate 0.88 - Stanford copy 0.12 Stanford . generate 0.95 - Note: Low p_gen indicates copying; high p_gen indicates generation
Handling Out-of-Vocabulary Words
The copy mechanism's most important contribution is handling out-of-vocabulary (OOV) words. Without copying, rare words would be replaced with <unk> tokens, destroying factual accuracy. With copying, these words can appear in the output even if they never occurred in training.
The implementation requires extending the vocabulary dynamically for each input. The key idea is that the model does not need to "know" a word in the traditional sense to reproduce it. It simply needs to be able to point to where that word appears in the source. This is a fundamental shift from the assumption that a model's capabilities are bounded by its training vocabulary. For OOV handling, the source document itself becomes a temporary vocabulary extension, one that is different for every input and does not require any additional learning.
Think of how a student handles an unknown technical term on an exam. If the term appears in the question, the student can copy it into their answer without understanding it. They simply recognize that the exam question contains a word that should appear in their answer, and they reproduce it. The pointer-generator does exactly this: it identifies tokens in the source that should be reproduced, and it copies them regardless of whether they appear in the training vocabulary.
The implementation uses a two-level vocabulary scheme. Standard tokens use their fixed vocabulary IDs. OOV tokens that appear in the source receive temporary "extended" IDs starting from the base vocabulary size. These extended IDs are local to each input example in the batch: OOV word 0 in one example may be a completely different word from OOV word 0 in another example. The extended IDs are only used during forward passes and loss computation; they are discarded afterward and recomputed for each new input.
class OOVHandler:
"""
Handles out-of-vocabulary words for copy mechanism.
"""
def __init__(self, vocab):
self.vocab = vocab
self.word_to_idx = {word: i for i, word in enumerate(vocab)}
self.vocab_size = len(vocab)
def process_source(self, source_tokens):
"""
Convert source tokens to IDs, tracking OOV words.
Returns:
source_ids: List of token IDs (OOV words get extended IDs)
oov_words: List of OOV words encountered
extended_vocab: Original vocab + OOV words
"""
source_ids = []
oov_words = []
oov_to_idx = {}
for token in source_tokens:
if token in self.word_to_idx:
source_ids.append(self.word_to_idx[token])
else:
# OOV word
if token not in oov_to_idx:
oov_to_idx[token] = len(oov_words)
oov_words.append(token)
# Extended ID = vocab_size + oov_index
source_ids.append(self.vocab_size + oov_to_idx[token])
extended_vocab = self.vocab + oov_words
return source_ids, oov_words, extended_vocab
def decode_output(self, output_ids, oov_words):
"""
Convert output IDs back to tokens, using OOV words when needed.
"""
output_tokens = []
for idx in output_ids:
if idx < self.vocab_size:
output_tokens.append(self.vocab[idx])
else:
oov_idx = idx - self.vocab_size
if oov_idx < len(oov_words):
output_tokens.append(oov_words[oov_idx])
else:
output_tokens.append("<unk>")
return output_tokens
# Example
base_vocab = [
"<pad>",
"<unk>",
"<sos>",
"<eos>",
"the",
"said",
"at",
"university",
]
handler = OOVHandler(base_vocab)
source = [
"Dr.",
"Chen",
"said",
"the",
"discovery",
"at",
"Stanford",
"university",
]
source_ids, oov_words, extended_vocab = handler.process_source(source)OOV Handling ================================================== Base vocabulary size: 8 Base vocab: ['<pad>', '<unk>', '<sos>', '<eos>', 'the', 'said', 'at', 'university'] Source: ['Dr.', 'Chen', 'said', 'the', 'discovery', 'at', 'Stanford', 'university'] Processed source IDs: [8, 9, 5, 4, 10, 6, 11, 7] OOV words found: ['Dr.', 'Chen', 'discovery', 'Stanford'] Extended vocabulary: ['<pad>', '<unk>', '<sos>', '<eos>', 'the', 'said', 'at', 'university', 'Dr.', 'Chen', 'discovery', 'Stanford'] Token mapping: Dr. -> 8 (OOV, extended) Chen -> 9 (OOV, extended) said -> 5 (in vocab) the -> 4 (in vocab) discovery -> 10 (OOV, extended) at -> 6 (in vocab) Stanford -> 11 (OOV, extended) university -> 7 (in vocab)
The OOV handler assigns extended vocabulary IDs to words not in the base vocabulary. "Dr.", "Chen", "discovery", and "Stanford" receive IDs starting from the base vocabulary size (8), allowing the copy mechanism to produce these tokens even though they weren't in the original vocabulary. This dynamic vocabulary extension is computed per-input, so different source documents can have different OOV words.
This per-input vocabulary creates a subtle complication at inference time. The model outputs an index into the extended vocabulary. For indices below vocab_size, the index maps to the standard vocabulary as usual. For indices above vocab_size, the model is pointing to an OOV word, and you need to look up which OOV word appeared for this particular input. The decode_output method above handles this mapping, converting extended indices back to the actual OOV strings using the list of OOV words that was built during preprocessing. This means the full pipeline for a single input involves preprocessing to identify OOV words and build extended IDs, a forward pass using those extended IDs, and postprocessing to convert extended IDs back to strings.
During training, we need to handle the case where target tokens are OOV. When a target token is OOV but appears in the source, we want the model to learn to copy it. This requires computing the loss against the extended vocabulary distribution: the model should assign high probability to the OOV token's extended ID, which can only happen through the copy channel. When a target token is OOV and does not appear in the source at all, there is truly nothing the model can do; it must fall back to <unk>, and the loss reflects this irreducible error.
def compute_loss_with_oov(final_dist, target_ids, vocab_size):
"""
Compute cross-entropy loss, handling OOV targets.
If target is OOV but copyable from source, loss is computed
against the extended distribution. If target is truly unknown,
it's mapped to <unk>.
"""
batch_size = target_ids.shape[0]
extended_size = final_dist.shape[1]
# Clamp target IDs to valid range
# OOV targets beyond extended vocab map to <unk> (index 1)
valid_targets = target_ids.clone()
valid_targets[valid_targets >= extended_size] = 1 # <unk>
# Gather probabilities for target tokens
target_probs = final_dist.gather(1, valid_targets.unsqueeze(1)).squeeze(1)
# Negative log likelihood
loss = -torch.log(target_probs + 1e-12)
return loss.mean()
# Example
target_ids = torch.tensor([8, 9, 5]) # "Dr.", "Chen", "said"
# "Dr." and "Chen" are OOV (ids 8, 9), "said" is in vocab (id 5)
# Simulated final distribution (extended vocab)
final_dist = torch.zeros(3, 12) # batch=3, extended_vocab=12
final_dist[0, 8] = 0.7 # High prob for "Dr." (copied)
final_dist[1, 9] = 0.6 # High prob for "Chen" (copied)
final_dist[2, 5] = 0.8 # High prob for "said" (generated)
loss = compute_loss_with_oov(final_dist, target_ids, len(base_vocab))Loss Computation with OOV
==================================================
Target tokens: ['Dr.', 'Chen', 'said']
Target IDs: [8, 9, 5]
Base vocab size: 8
Probabilities assigned to targets:
P('Dr.') = 0.70 (OOV, copied)
P('Chen') = 0.60 (OOV, copied)
P('said') = 0.80 (in vocab, generated)
Loss: 0.3635Attention Visualization for Copy
Visualizing attention patterns reveals when and what the model copies. High attention on specific source positions often indicates copying, especially when is low.
The visual signature of copying is distinctive: when the model copies a token, attention concentrates sharply on the corresponding source position, often with more than 80% of the probability mass on a single position. When the model generates a token, attention is typically more diffuse, spread across several source positions that provide context without any single position being the "source" of the output token. This difference in attention sharpness is not just visually useful; it can be used diagnostically to understand what the model is doing at each step, and it has been used in research to study how well the model's copy decisions align with human judgments about which tokens should be copied.
Coverage Mechanism
One issue with attention-based models is repetition: the model may attend to the same source positions multiple times, generating repetitive output. The coverage mechanism addresses this by tracking which source positions have already been attended to.
Repetition is a particularly visible failure mode in summarization. Models without coverage mechanisms have a tendency to produce summaries like "The president announced a new plan. The president announced the plan. The plan was announced by the president." This happens because, at each decoding step, the attention mechanism makes decisions based only on the current decoder state and the encoder outputs, with no memory of what it has already attended to. If the decoder state happens to be in a configuration that points attention toward "the president announced," it will generate that phrase again, because nothing in the standard attention formulation prevents it. The coverage mechanism adds exactly this memory: a record of where attention has already been focused.
Coverage maintains a running sum of attention distributions from all previous decoder steps. This coverage vector is used to penalize re-attending to already-covered positions, reducing repetition in the output.
The coverage vector at decoder step accumulates all previous attention distributions:
where:
- : the coverage vector at step , with one value per source position
- : the attention distribution at previous step
- The sum runs over all previous decoder steps from to
Each element represents how much total attention has been paid to source position so far. High values indicate positions that have been heavily attended; low values indicate under-attended positions.
The coverage vector serves two roles. First, it is fed as an additional input to the attention scoring function. This allows the current attention distribution to be conditioned on the coverage history. The attention mechanism learns to lower scores for already-covered positions, effectively discouraging re-attendance through the scoring mechanism itself. Second, a separate coverage loss directly penalizes re-attending to covered positions during training. This provides a stronger gradient signal to reinforce the desired behavior.
This coverage vector is incorporated into the attention computation, encouraging the model to attend to positions with low coverage. Additionally, a coverage loss explicitly penalizes re-attending to already-covered positions during training:
where:
- : the coverage loss at step
- : the current attention weight on source position
- : the accumulated coverage at position
- : takes the element-wise minimum
The intuition behind the function: the loss is only incurred when both the current attention and the past coverage are high for the same position. If either is low, the contribution to the loss is small. This allows the model to attend to new positions freely while penalizing redundant attention to already-covered content.
the formulation is captures more detail than a simple product would be. Consider position with past coverage and current attention . The product would be , still moderately penalizing the small current attention. But the gives . This reflects that the current attention is the binding constraint: since the model is only attending to this position a little, the redundancy penalty is proportionally small. Conversely, if and , you are attending heavily to a position that was barely covered before, which is desirable behavior. The gives , a small penalty, which is correct. The thus charges a penalty proportional to the smaller of the two quantities, which precisely captures the intuition that clear redundancy requires clear weight in both the current step and the past history.
class CoverageAttention(nn.Module):
"""
Attention with coverage mechanism to reduce repetition.
"""
def __init__(self, hidden_dim):
super().__init__()
self.hidden_dim = hidden_dim
# Standard attention parameters
self.W_encoder = nn.Linear(hidden_dim, hidden_dim, bias=False)
self.W_decoder = nn.Linear(hidden_dim, hidden_dim, bias=False)
self.W_coverage = nn.Linear(1, hidden_dim, bias=False)
self.v = nn.Linear(hidden_dim, 1, bias=False)
def forward(self, decoder_state, encoder_outputs, coverage, mask=None):
"""
Compute attention with coverage.
Args:
decoder_state: (batch, hidden)
encoder_outputs: (batch, seq_len, hidden)
coverage: (batch, seq_len) - sum of previous attention weights
mask: (batch, seq_len)
Returns:
attn_weights: (batch, seq_len)
coverage_loss: scalar
"""
batch_size, seq_len, _ = encoder_outputs.shape
# Project inputs
encoder_proj = self.W_encoder(encoder_outputs) # (batch, seq, hidden)
decoder_proj = self.W_decoder(decoder_state).unsqueeze(
1
) # (batch, 1, hidden)
coverage_proj = self.W_coverage(
coverage.unsqueeze(-1)
) # (batch, seq, hidden)
# Compute scores with coverage
scores = self.v(torch.tanh(encoder_proj + decoder_proj + coverage_proj))
scores = scores.squeeze(-1) # (batch, seq_len)
if mask is not None:
scores = scores.masked_fill(~mask, float("-inf"))
attn_weights = F.softmax(scores, dim=-1)
# Coverage loss: penalize re-attending
coverage_loss = torch.sum(
torch.min(attn_weights, coverage), dim=-1
).mean()
return attn_weights, coverage_loss
# Example
# Use dimensions from PointerGeneratorDecoder
cov_hidden_dim = 128
cov_batch_size = 2
cov_src_len = 10
coverage_attn = CoverageAttention(cov_hidden_dim)
# Create matching tensors
cov_decoder_state = torch.randn(cov_batch_size, cov_hidden_dim)
cov_encoder_outputs = torch.randn(cov_batch_size, cov_src_len, cov_hidden_dim)
# Simulated coverage from previous steps
coverage = torch.tensor([[0.0, 0.3, 0.5, 0.1, 0.1, 0.0, 0.0, 0.0, 0.0, 0.0]])
coverage = coverage.expand(cov_batch_size, -1)
attn_weights, cov_loss = coverage_attn(
cov_decoder_state,
cov_encoder_outputs,
coverage,
)Coverage Mechanism ================================================== Previous coverage (positions already attended): Position 1: 0.30 ██████ Position 2: 0.50 ██████████ Position 3: 0.10 ██ Position 4: 0.10 ██ New attention weights: Position 0: 0.099 █ Position 1: 0.087 █ Position 2: 0.101 ██ Position 3: 0.087 █ Position 4: 0.101 ██ Position 5: 0.154 ███ Position 6: 0.124 ██ Position 7: 0.088 █ Position 8: 0.076 █ Position 9: 0.083 █ Coverage loss: 0.3852 (Lower is better - means less re-attending to covered positions)
The visualization shows how coverage accumulates during generation. In the left panel, each row represents the attention distribution at a single decoding step. The right panel shows the cumulative coverage before each step. Notice how "president" (position 1) quickly accumulates high coverage after step 0, discouraging the model from re-attending to it. By step 4, the coverage mechanism has effectively "used up" the early positions, encouraging the model to attend to later, less-covered positions.
The coverage mechanism illustrates a broader principle in sequence modeling: sometimes the right way to improve a model is not to add more capacity but to add the right inductive bias. A standard LSTM decoder theoretically has enough capacity to remember what it has already generated and avoid repetition, but in practice it does not reliably learn to do so from the training signal alone. The coverage vector explicitly encodes this information in a structured way, making it easy for the attention mechanism to condition on coverage history. This kind of structured auxiliary information often produces better and more sample-efficient learning than hoping the model will discover the same structure implicitly.
Practical Training Considerations
Training pointer-generator networks requires careful attention to several practical details. The architecture introduces several interacting components, each with its own training dynamics, and getting them to work together reliably requires understanding both the mechanics and the common failure modes.
Teacher forcing with copy targets: During training, when the target token appears in the source, the model should learn to copy it. This requires computing whether each target token is copyable and adjusting the loss accordingly. The standard cross-entropy loss applies to the extended vocabulary distribution: if the target is an OOV token that appears in the source, the model should learn to assign high probability to its extended vocabulary ID, which it can only do through the copy channel. This implicitly trains the copy mechanism to activate for OOV tokens without requiring explicit copy labels.
def prepare_copy_targets(source_ids, target_ids, vocab_size):
"""
Prepare targets for copy-aware training.
Returns mask indicating which target tokens are copyable from source.
"""
batch_size, tgt_len = target_ids.shape
_, src_len = source_ids.shape
# For each target position, check if token appears in source
copyable = torch.zeros(batch_size, tgt_len, dtype=torch.bool)
copy_positions = torch.zeros(batch_size, tgt_len, src_len)
for b in range(batch_size):
for t in range(tgt_len):
tgt_token = target_ids[b, t].item()
for s in range(src_len):
if source_ids[b, s].item() == tgt_token:
copyable[b, t] = True
copy_positions[b, t, s] = 1.0
# Normalize copy positions
copy_positions = copy_positions / (
copy_positions.sum(dim=-1, keepdim=True) + 1e-12
)
return copyable, copy_positions
# Example
source_ids = torch.tensor(
[[5, 8, 9, 4, 10]]
) # "the Chen Stanford said discovery"
target_ids = torch.tensor([[8, 11, 10]]) # "Chen announced discovery"
copyable, copy_pos = prepare_copy_targets(source_ids, target_ids, vocab_size=20)Copy Target Preparation
==================================================
Source IDs: [5, 8, 9, 4, 10]
Target IDs: [8, 11, 10]
Target token analysis:
Chen: copyable
Copy from source position(s): [1]
announced: must generate
discovery: copyable
Copy from source position(s): [4]The prepare_copy_targets function above computes two useful outputs. The copyable mask identifies which target positions can be satisfied through copying, which is useful for analysis and debugging. The copy_positions tensor identifies exactly which source positions contain each target token, which can be used to compute an auxiliary "copy alignment" loss that explicitly trains the attention mechanism to concentrate on the right source positions when copying. This auxiliary loss is optional but can speed up early training when the copy mechanism is still learning to associate target tokens with their source positions.
Additional training techniques include:
- Scheduled sampling for : Early in training, the model may not learn when to copy effectively. Some implementations initialize with a bias toward copying for OOV tokens, then let learn freely once the basic copy mechanism is working.
- Gradient clipping: The combined distribution can have very small probabilities for OOV tokens early in training, leading to large gradients through the loss. Gradient clipping at a norm of 2.0 to 5.0 helps stabilize training during the early phases.
- Coverage loss weighting: The coverage loss is added to the main cross-entropy loss with a weighting coefficient, typically tuned between 0.5 and 2.0. Too high a weight makes the model too aggressive at avoiding repetition, which can hurt fluency. Too low a weight and the coverage mechanism has insufficient effect. A common practice is to start with no coverage loss, train until convergence, then fine-tune with coverage loss added.
- Batch construction: Since extended vocabulary sizes vary with the source document length and OOV count, batching requires padding both the source sequences and the extended vocabulary distributions to a uniform size. A maximum OOV count per batch (typically 50 to 200) prevents the extended vocabulary from growing excessively large for batches containing very long or vocabulary-diverse documents.
The boxplot reveals the learned specialization that emerges from training without any explicit annotation of which tokens should be copied. Function words like "the," "of," and "in" cluster near 1.0 because the model learns they are never worth copying: they are far more reliably generated from the vocabulary than pointed to in the source. Numbers and named entities cluster near 0.0 because the model learns that whenever they appear in the target, the safest strategy is to copy them directly from the source. Content words occupy the most interesting middle ground, where context determines the optimal strategy and the distribution is correspondingly wide.
Limitations and Impact
The copy mechanism, while powerful, has important limitations that practitioners should understand.
The mechanism assumes that words worth copying appear verbatim in the source. For tasks requiring paraphrasing or where source words need morphological changes (e.g., "announced" to "announcement"), pure copying falls short. The model must learn to generate these variants, which may still be OOV. Some extensions address this by copying at the subword level, allowing partial matches and morphological flexibility. If you tokenize both source and target using byte-pair encoding or WordPiece, OOV words are decomposed into familiar subword units, which largely sidesteps the original OOV problem. This is one reason why modern transformer-based models, which use subword tokenization universally, rarely need explicit copy mechanisms for the OOV problem specifically.
Copy mechanisms add computational overhead. Computing the extended vocabulary distribution and tracking OOV words increases memory usage and slows inference. For very long source documents, the attention computation over all source positions becomes expensive. A 1,000-word document requires computing and storing attention weights over 1,000 positions at each decoding step, which is quadratic in document length. Hierarchical attention and sparse attention variants can mitigate this, but at the cost of additional complexity and potential coverage gaps where relevant information might be missed.
The balance between copying and generating is learned implicitly through . In practice, models sometimes over-copy, producing extractive rather than abstractive summaries where long phrases are copied verbatim instead of being paraphrased. This over-copying can arise when the training data contains many reference summaries that closely mirror the source text, teaching the model that copying is the safe default strategy. Conversely, models can under-copy when trained on highly abstractive reference summaries, learning to paraphrase even when a direct copy would be more accurate. Careful curation of training data and tuning of the coverage loss weight are often necessary to achieve the right balance, but this requires domain expertise and experimentation.
The copy mechanism also inherits any errors present in the source document. If the source contains a misspelling or factual error, the model may faithfully copy it into the output. This is a fundamental limitation of the approach: the model can only be as accurate as its source material. For applications where the source may contain errors, additional mechanisms to cross-reference multiple sources or to verify copied facts would be needed.
Despite these limitations, copy mechanisms significantly improved neural text generation. Before their introduction, neural summarization systems struggled with factual accuracy, producing fluent but unfaithful summaries where names became generic placeholders and numbers disappeared entirely. The pointer-generator architecture demonstrated that neural models could preserve factual content while still generating abstractive summaries, achieving a level of faithfulness that purely generative models could not match. The CNN/DailyMail results from See et al. (2017) were substantially better than prior work on both automatic metrics (ROUGE) and human evaluation of factual accuracy.
The influence of copy mechanisms extends beyond summarization. Question answering systems adapted pointer networks to extract answer spans directly from context passages, a design that became foundational in models like BiDAF and the original DrQA. Dialogue systems used copy mechanisms to refer back to entities mentioned earlier in the conversation, improving coherence in multi-turn interactions. Machine translation incorporated copying to handle named entities and technical terms that should not be translated but preserved. Each of these adaptations reflects the same core insight: when some output tokens should come from the input rather than the vocabulary, giving the model a direct pathway for that operation improves both accuracy and efficiency.
The connection to retrieval-augmented generation is also direct. Modern RAG systems retrieve relevant passages and condition generation on them, which is structurally similar to what a pointer-generator does at a coarser level. The copy mechanism treats the entire source document as a retrievable resource; RAG systems treat a retrieved set of passages similarly. The key difference is that RAG typically relies on large language models with massive vocabularies and strong in-context learning abilities to handle the copy-or-generate decision implicitly, rather than through explicit architectural machinery. Understanding the explicit copy mechanism provides insight into what those large models are doing implicitly when they reproduce facts from their context.
Summary
Copy mechanisms extend sequence-to-sequence models to handle the fundamental vocabulary limitation of neural text generation. The key concepts from this chapter:
-
Pointer networks use attention weights as a probability distribution over input positions, enabling the model to "point to" and copy input tokens rather than generating from a fixed vocabulary. The key insight is that attention already computes a relevance score for each input position; pointer networks simply interpret those scores as selection probabilities.
-
The generation probability acts as a soft switch between generating from vocabulary and copying from input. It is computed from the decoder state, context vector, and input embedding, allowing the model to learn when each strategy is appropriate without any explicit supervision about copy decisions.
-
The final distribution combines generation and copy probabilities through a weighted mixture. Words appearing in both vocabulary and input receive probability from both sources, while OOV words can only be produced through copying. The mathematical structure ensures the output remains a valid probability distribution.
-
Pointer-generator networks integrate these components into a practical architecture for tasks like summarization, where preserving names, numbers, and rare words is essential for factual accuracy. The architecture adds minimal overhead to a standard seq2seq model.
-
OOV handling requires extending the vocabulary dynamically for each input, tracking which source positions contain which OOV words, and ensuring the loss function properly handles extended vocabulary targets. The extended vocabulary is ephemeral: it exists only for the duration of processing a single input.
-
Coverage mechanisms address repetition by tracking cumulative attention over all previous decoder steps and penalizing re-attention to already-covered positions. The coverage loss uses a function that incurs penalty only when both current attention and past coverage are high for the same position.
The copy mechanism was an important step in making neural text generation practical for real-world applications where factual accuracy matters. While modern large language models have largely subsumed these techniques through massive subword vocabularies and in-context learning, understanding copy mechanisms provides insight into the fundamental challenges of neural text generation and the principled solutions researchers developed. The explicit architectural choices made in the pointer-generator, separating copy from generate, maintaining coverage, extending vocabulary dynamically, illuminate problems that large models solve implicitly at much greater scale. When you see a language model faithfully reproduce a fact from its context window, the underlying computational process is a sophisticated learned approximation of the same copy-or-generate decision this chapter has made explicit.
Key Parameters
When implementing pointer-generator networks:
Copy Switch Parameters:
hidden_dim: Dimension of encoder/decoder hidden states. Larger values capture more detailed representations but increase computation. Typical values: 256-512 for small models, 512-1024 for larger ones.embed_dim: Dimension of token embeddings fed to the copy switch. Should match the embedding layer used in the decoder.
Attention Parameters:
hidden_dimin attention: Must match encoder output dimension. The attention mechanism projects both encoder and decoder states to this dimension for score computation.
Coverage Parameters:
coverage_loss_weight: Hyperparameter controlling how strongly to penalize re-attending to covered positions. Values between 0.5 and 2.0 are common. Higher values reduce repetition more aggressively but may hurt fluency.
Training Parameters:
max_oov_per_batch: Maximum number of OOV words to track per batch. Limits memory usage for the extended vocabulary. Typical values: 50-200 depending on document length.gradient_clip: Maximum gradient norm for clipping. Values around 2.0-5.0 help stabilize training when probabilities become very small.
Quiz
Ready to test your understanding? Take this quick quiz to reinforce what you've learned about copy mechanisms in neural text generation.
Copy Mechanism Quiz
Reference
Citation details
Cite or share this article.
Continue with the full handbook
This chapter is part of Language AI Handbook. Use the handbook page to browse the complete table of contents and continue reading in sequence.
Explore Language AI HandbookStay up to date
Get articles, book updates, and news delivered to your inbox.
No spam, unsubscribe anytime.
Join the community
Sign in to remove popups, track your reading progress, and join the discussion.
Comments
No comments yet. Be the first to share your thoughts!