Copy Mechanism › Pointer Networks: The Foundation
Language AI HandbookAll chapters Copy Mechanism › Pointer Networks: The FoundationOne section · Sections in this chapter3 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
To understand copy mechanisms, we must first grapple with a fundamental question: how can a neural network select items from a variable-length input sequence? Traditional neural networks produce outputs of fixed size, but when copying from input, we need to point to one of positions where changes with each input.
Pointer networks, introduced by Vinyals et al. in 2015, solve this by repurposing attention. Recall from earlier chapters that attention computes a weighted combination of encoder states, producing weights that sum to 1 across all input positions. These weights already form a probability distribution over the input. The key insight is simple: instead of using attention weights only to compute context vectors, we can interpret them directly as "pointing" probabilities.
Pointer networks (Vinyals et al., 2015) were originally developed not for text generation but for combinatorial optimization problems like the Traveling Salesman Problem, where the output must be a permutation of the input. The model needed to output sequences like "visit city 3, then city 1, then city 5," selecting from a set of cities whose size varied between problem instances. Standard softmax over a fixed vocabulary could not handle this because the set of cities changes with each input. The pointer network's solution, treating attention weights as selection probabilities, turned out to be equally applicable to text generation, where the "cities" are input tokens.
The original motivation from combinatorial optimization is instructive. In the TSP setting, the vocabulary is literally the input itself: the model must output pointers into the same sequence it reads. This framing makes it obvious that attention weights are the right abstraction, because attention already computes a relevance score for each input position. In text generation, the insight carries over: the source tokens form a dynamic vocabulary, and the attention distribution is already computing something very close to "how relevant is each source token right now?" Pointer networks simply formalize this connection and use it directly as a selection mechanism.
What makes pointer networks elegant is that they add almost nothing to a standard attention-based seq2seq model. The attention mechanism was already there; the pointer network just changes how you interpret its output. Instead of multiplying attention weights by encoder states to get a context vector, you use those same weights directly as a probability distribution over which input position to point to. The architectural change is minimal, but the capability it adds is substantial.
A pointer network is a sequence-to-sequence model where the output at each step is a pointer to an element in the input sequence. Instead of generating tokens from a vocabulary, it uses attention over the input to select which input position to "point to."
Let's trace through the mechanics. At each decoder step, we have a decoder hidden state that encodes what we've generated so far. We also have encoder outputs representing each input position. The pointer mechanism computes a score for each input position, measuring how relevant that position is given the current decoder state:
- Project both representations into a common space using learned weight matrices
- Combine them through addition (Bahdanau-style) and pass through a nonlinearity
- Compute a scalar score for each position using a learned vector
- Normalize with softmax to obtain a valid probability distribution
The resulting probabilities tell us: "Given what I've generated so far, how likely should I point to each input position?"
In practice, the scoring function is the same Bahdanau additive attention you encountered in the attention chapter. The difference is purely in how the output is used. In standard attention, the weights are intermediate values used to compute a context vector. In a pointer network, the weights are the final output: they are the model's answer to the question "which input token should I select?" This reuse of existing machinery is why pointer networks can be integrated into seq2seq models with minimal architectural overhead.
import torch
import torch.nn as nn
import torch.nn.functional as F
class PointerAttention(nn.Module):
"""
Attention mechanism that produces pointer probabilities.
"""
def __init__(self, hidden_dim):
super().__init__()
self.hidden_dim = hidden_dim
# Attention parameters (Bahdanau-style)
self.W_encoder = nn.Linear(hidden_dim, hidden_dim, bias=False)
self.W_decoder = nn.Linear(hidden_dim, hidden_dim, bias=False)
self.v = nn.Linear(hidden_dim, 1, bias=False)
def forward(self, decoder_state, encoder_outputs, mask=None):
"""
Compute pointer probabilities over input positions.
Args:
decoder_state: (batch, hidden_dim)
encoder_outputs: (batch, seq_len, hidden_dim)
mask: (batch, seq_len) - True for valid positions
Returns:
pointer_probs: (batch, seq_len) - probability of pointing to each input
"""
batch_size, seq_len, _ = encoder_outputs.shape
# Project encoder outputs: (batch, seq_len, hidden)
encoder_proj = self.W_encoder(encoder_outputs)
# Project decoder state: (batch, hidden) -> (batch, 1, hidden)
decoder_proj = self.W_decoder(decoder_state).unsqueeze(1)
# Compute attention scores: (batch, seq_len, 1) -> (batch, seq_len)
scores = self.v(torch.tanh(encoder_proj + decoder_proj)).squeeze(-1)
# Apply mask (set invalid positions to -inf)
if mask is not None:
scores = scores.masked_fill(~mask, float("-inf"))
# Softmax to get pointer probabilities
pointer_probs = F.softmax(scores, dim=-1)
return pointer_probs
# Create example
hidden_dim = 64
batch_size = 2
seq_len = 5
attention = PointerAttention(hidden_dim)
decoder_state = torch.randn(batch_size, hidden_dim)
encoder_outputs = torch.randn(batch_size, seq_len, hidden_dim)
mask = torch.ones(batch_size, seq_len, dtype=torch.bool)
pointer_probs = attention(decoder_state, encoder_outputs, mask)Pointer Attention Output ================================================== Input sequence length: 5 Pointer probabilities (batch 0): Position 0: 0.161 ████ Position 1: 0.243 ███████ Position 2: 0.214 ██████ Position 3: 0.163 ████ Position 4: 0.218 ██████ Sum of probabilities: 1.0000
The pointer probabilities form a valid distribution over input positions. At each decoding step, the model can select which input token to copy by sampling from or taking the argmax of this distribution.
Notice that the sum of pointer probabilities equals exactly 1.0. This is a direct consequence of the softmax normalization, and it is an important property: it means we can interpret these values as valid probabilities and mix them with other probability distributions in a mathematically principled way, which is exactly what the pointer-generator network will do in the next section.