Copy Mechanism › The Vocabulary Problem in Generation
Language AI HandbookAll chapters Copy Mechanism › The Vocabulary Problem in GenerationOne section · Sections in this chapter2 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Before diving into copy mechanisms, let's understand the problem they solve. Traditional seq2seq models have a fixed output vocabulary, typically the most frequent words in the training data. This design choice is almost inevitable: the final softmax layer must produce a probability over some finite set of tokens, and the practical size of that set is constrained by memory and the need for sufficient training examples per token. A vocabulary of 50,000 words covers the vast majority of everyday language, but it covers only a small fraction of all the proper nouns, technical terms, product names, and numerical values that appear in real-world text.
The problem is asymmetric: a model trained on a large corpus will have no difficulty generating words like "the," "announced," or "discovery," but it will fail entirely on words like "Nakamura," "CRISPR," or "Q3-FY2024." These words may appear once or twice in the training data, not enough to learn a reliable embedding, or they may never appear at all if they postdate the training cutoff. When the model encounters such words at inference time, it has no representation for them. The only option the standard vocabulary distribution offers is the catch-all <unk> token, which carries no information about the actual word that should be there.
from collections import Counter
# Simulate a typical vocabulary scenario
training_corpus = """
The president announced new policies today.
Scientists discovered a breakthrough in medicine.
The company reported strong quarterly earnings.
Researchers published findings in Nature journal.
The government proposed infrastructure spending.
"""
# Build vocabulary from training data
words = training_corpus.lower().split()
word_counts = Counter(words)
# Typical vocab: keep only frequent words
vocab_size = 20
vocab = ["<unk>", "<sos>", "<eos>"] + [
word for word, _ in word_counts.most_common(vocab_size - 3)
]
word_to_idx = {word: i for i, word in enumerate(vocab)}Vocabulary Construction ================================================== Vocabulary size: 20 Top words in vocab: <unk> <sos> <eos> the in president announced new policies today. ...
Now consider what happens when we encounter input with rare words:
# Input with rare/unseen words
test_input = "Dr. Nakamura presented findings at the Stanford conference."
# Check which words are in vocabulary
in_vocab = []
out_of_vocab = []
for word in test_input.lower().replace(".", "").split():
if word in word_to_idx:
in_vocab.append(word)
else:
out_of_vocab.append(word)Out-of-Vocabulary Analysis ================================================== Input: Dr. Nakamura presented findings at the Stanford conference. In vocabulary: ['the'] Out of vocabulary (OOV): ['dr', 'nakamura', 'presented', 'findings', 'at', 'stanford', 'conference'] OOV rate: 87.5%
The words "Nakamura," "Stanford," and "conference" are critical for an accurate summary, yet a standard decoder cannot produce them. It would have to replace them with <unk> tokens or substitute similar but incorrect words. Copy mechanisms solve this directly: let the decoder point to these words in the input and copy them.
This capability directly affects factual accuracy. For tasks like question answering, summarization, and dialogue, names and numbers must be reproduced exactly. Replacing "Stanford" with "the university" or "Nakamura" with a <unk> token defeats the system.s purpose. The OOV problem is especially severe for named entities, which are precisely the words that carry the most information and are the most difficult to recover from context alone.