Copy Mechanism › The Pointer-Generator Network
Language AI HandbookAll chapters Copy Mechanism › The Pointer-Generator NetworkOne section · Sections in this chapter6 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The pointer-generator network, introduced by See et al. (2017) for abstractive summarization, combines the ideas above into a cohesive architecture. It extends the standard attention-based seq2seq model with a copy mechanism, enabling it to both generate words from the vocabulary and copy words from the source document.
The 2017 paper by Abigail See, Peter Liu, and Christopher Manning was a landmark in neural summarization research. Their model was evaluated on the CNN/DailyMail dataset, a standard benchmark where news articles are summarized using bullet-point highlights. The pointer-generator model substantially outperformed prior neural approaches on ROUGE metrics, and more importantly, it produced summaries with dramatically fewer factual errors because it could copy names, numbers, and entities directly from the source. The paper also introduced the coverage mechanism discussed later in this chapter, addressing the separate problem of repetitive output. Together, these two contributions made the pointer-generator the go-to architecture for neural summarization for several years.
The architecture is elegant precisely because it adds so little to the baseline. The encoder is a standard bidirectional LSTM that processes the source document. The decoder is a standard LSTM with attention. The entire copy mechanism adds only two components: the copy switch (a small linear layer plus sigmoid) and the distribution combination step. The extended vocabulary and OOV tracking add some bookkeeping complexity, but no additional trainable parameters beyond those in the switch. This efficiency was a significant practical advantage when the alternative was large, complex architectures that were difficult to train.
Let's implement a complete pointer-generator decoder step:
class PointerGeneratorDecoder(nn.Module):
"""
Single decoding step of a pointer-generator network.
"""
def __init__(self, vocab_size, embed_dim, hidden_dim):
super().__init__()
self.vocab_size = vocab_size
self.hidden_dim = hidden_dim
# Embedding and LSTM
self.embedding = nn.Embedding(vocab_size, embed_dim)
self.lstm = nn.LSTMCell(embed_dim + hidden_dim, hidden_dim)
# Attention
self.attention = PointerAttention(hidden_dim)
# Copy switch
self.copy_switch = CopySwitch(hidden_dim, embed_dim)
# Vocabulary projection
self.vocab_proj = nn.Linear(hidden_dim * 2, vocab_size)
def forward(
self,
input_token,
prev_hidden,
prev_cell,
encoder_outputs,
source_ids,
encoder_mask=None,
):
"""
Perform one decoding step.
Args:
input_token: (batch,) - previous output token
prev_hidden: (batch, hidden) - previous hidden state
prev_cell: (batch, hidden) - previous cell state
encoder_outputs: (batch, src_len, hidden) - encoder outputs
source_ids: (batch, src_len) - source token IDs
encoder_mask: (batch, src_len) - valid source positions
Returns:
final_dist: (batch, extended_vocab) - output distribution
hidden: (batch, hidden) - new hidden state
cell: (batch, hidden) - new cell state
attn_weights: (batch, src_len) - attention weights
"""
batch_size = input_token.shape[0]
# Embed input
embedded = self.embedding(input_token) # (batch, embed)
# Compute attention over encoder outputs
attn_weights = self.attention(
prev_hidden, encoder_outputs, encoder_mask
)
# Context vector
context = torch.bmm(attn_weights.unsqueeze(1), encoder_outputs).squeeze(
1
)
# LSTM input: concatenate embedding and context
lstm_input = torch.cat([embedded, context], dim=-1)
# LSTM step
hidden, cell = self.lstm(lstm_input, (prev_hidden, prev_cell))
# Vocabulary distribution
vocab_input = torch.cat([hidden, context], dim=-1)
vocab_logits = self.vocab_proj(vocab_input)
vocab_dist = F.softmax(vocab_logits, dim=-1)
# Copy probability
p_gen = self.copy_switch(context, hidden, embedded)
# Combine distributions
final_dist = self._combine_distributions(
p_gen, vocab_dist, attn_weights, source_ids
)
return final_dist, hidden, cell, attn_weights
def _combine_distributions(
self, p_gen, vocab_dist, attn_weights, source_ids
):
"""Combine generation and copy distributions."""
batch_size, src_len = source_ids.shape
# For simplicity, assume no OOV (would need extended vocab otherwise)
final_dist = p_gen * vocab_dist
# Add copy probabilities
p_copy = 1 - p_gen
# Use scatter_add for efficiency
copy_dist = torch.zeros_like(vocab_dist)
copy_dist.scatter_add_(1, source_ids, attn_weights * p_copy)
final_dist = final_dist + copy_dist
return final_dist
# Test the decoder
vocab_size = 1000
embed_dim = 64
hidden_dim = 128
batch_size = 2
src_len = 10
decoder = PointerGeneratorDecoder(vocab_size, embed_dim, hidden_dim)
# Dummy inputs
input_token = torch.randint(0, vocab_size, (batch_size,))
prev_hidden = torch.zeros(batch_size, hidden_dim)
prev_cell = torch.zeros(batch_size, hidden_dim)
encoder_outputs = torch.randn(batch_size, src_len, hidden_dim)
source_ids = torch.randint(0, vocab_size, (batch_size, src_len))
final_dist, hidden, cell, attn = decoder(
input_token, prev_hidden, prev_cell, encoder_outputs, source_ids
)Pointer-Generator Decoder Output ================================================== Output distribution shape: torch.Size([2, 1000]) Hidden state shape: torch.Size([2, 128]) Attention weights shape: torch.Size([2, 10]) Top 5 predicted tokens (batch 0): Token 361: 0.0683 Token 101: 0.0669 Token 621: 0.0636 Token 86: 0.0592 Token 645: 0.0568 Distribution sum: 1.0000
The output distribution sums to 1.0, confirming it's a valid probability distribution. The decoder produces probabilities over all tokens in the vocabulary, and the top predictions show which tokens are most likely at this step. In practice, these probabilities would be used either for greedy decoding (selecting the argmax) or for beam search (maintaining multiple hypotheses).
The implementation above uses scatter_add_ for the copy distribution, which is an efficient tensor operation that accumulates copy probabilities into the right vocabulary positions in a single pass. This is important for performance: the naive loop over source positions that the compute_final_distribution function earlier used works for illustration purposes but would be far too slow for real-world documents with hundreds or thousands of tokens. The scatter operation achieves the same result in vectorized form, making it suitable for batched training.