Copy Mechanism › Summary

Copy mechanisms extend sequence-to-sequence models to handle the fundamental vocabulary limitation of neural text generation. The key concepts from this chapter:

• Pointer networks use attention weights as a probability distribution over input positions, enabling the model to "point to" and copy input tokens rather than generating from a fixed vocabulary. The key insight is that attention already computes a relevance score for each input position; pointer networks simply interpret those scores as selection probabilities.

• The generation probability $p_{\text{gen}}$ acts as a soft switch between generating from vocabulary and copying from input. It is computed from the decoder state, context vector, and input embedding, allowing the model to learn when each strategy is appropriate without any explicit supervision about copy decisions.

• The final distribution combines generation and copy probabilities through a weighted mixture. Words appearing in both vocabulary and input receive probability from both sources, while OOV words can only be produced through copying. The mathematical structure ensures the output remains a valid probability distribution.

• Pointer-generator networks integrate these components into a practical architecture for tasks like summarization, where preserving names, numbers, and rare words is essential for factual accuracy. The architecture adds minimal overhead to a standard seq2seq model.

• OOV handling requires extending the vocabulary dynamically for each input, tracking which source positions contain which OOV words, and ensuring the loss function properly handles extended vocabulary targets. The extended vocabulary is ephemeral: it exists only for the duration of processing a single input.

• Coverage mechanisms address repetition by tracking cumulative attention over all previous decoder steps and penalizing re-attention to already-covered positions. The coverage loss uses a $\min$ function that incurs penalty only when both current attention and past coverage are high for the same position.

The copy mechanism was an important step in making neural text generation practical for real-world applications where factual accuracy matters. While modern large language models have largely subsumed these techniques through massive subword vocabularies and in-context learning, understanding copy mechanisms provides insight into the fundamental challenges of neural text generation and the principled solutions researchers developed. The explicit architectural choices made in the pointer-generator, separating copy from generate, maintaining coverage, extending vocabulary dynamically, illuminate problems that large models solve implicitly at much greater scale. When you see a language model faithfully reproduce a fact from its context window, the underlying computational process is a sophisticated learned approximation of the same copy-or-generate decision this chapter has made explicit.

When implementing pointer-generator networks:

Copy Switch Parameters:

• hidden_dim: Dimension of encoder/decoder hidden states. Larger values capture more detailed representations but increase computation. Typical values: 256-512 for small models, 512-1024 for larger ones.
• embed_dim: Dimension of token embeddings fed to the copy switch. Should match the embedding layer used in the decoder.

Attention Parameters:

• hidden_dim in attention: Must match encoder output dimension. The attention mechanism projects both encoder and decoder states to this dimension for score computation.

Coverage Parameters:

• coverage_loss_weight: Hyperparameter controlling how strongly to penalize re-attending to covered positions. Values between 0.5 and 2.0 are common. Higher values reduce repetition more aggressively but may hurt fluency.

Training Parameters:

• max_oov_per_batch: Maximum number of OOV words to track per batch. Limits memory usage for the extended vocabulary. Typical values: 50-200 depending on document length.
• gradient_clip: Maximum gradient norm for clipping. Values around 2.0-5.0 help stabilize training when probabilities become very small.