Copy Mechanism › Handling Out-of-Vocabulary Words
Language AI HandbookAll chapters Copy Mechanism › Handling Out-of-Vocabulary WordsOne section · Sections in this chapter8 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The copy mechanism's most important contribution is handling out-of-vocabulary (OOV) words. Without copying, rare words would be replaced with <unk> tokens, destroying factual accuracy. With copying, these words can appear in the output even if they never occurred in training.
The implementation requires extending the vocabulary dynamically for each input. The key idea is that the model does not need to "know" a word in the traditional sense to reproduce it. It simply needs to be able to point to where that word appears in the source. This is a fundamental shift from the assumption that a model's capabilities are bounded by its training vocabulary. For OOV handling, the source document itself becomes a temporary vocabulary extension, one that is different for every input and does not require any additional learning.
Think of how a student handles an unknown technical term on an exam. If the term appears in the question, the student can copy it into their answer without understanding it. They simply recognize that the exam question contains a word that should appear in their answer, and they reproduce it. The pointer-generator does exactly this: it identifies tokens in the source that should be reproduced, and it copies them regardless of whether they appear in the training vocabulary.
The implementation uses a two-level vocabulary scheme. Standard tokens use their fixed vocabulary IDs. OOV tokens that appear in the source receive temporary "extended" IDs starting from the base vocabulary size. These extended IDs are local to each input example in the batch: OOV word 0 in one example may be a completely different word from OOV word 0 in another example. The extended IDs are only used during forward passes and loss computation; they are discarded afterward and recomputed for each new input.
class OOVHandler:
"""
Handles out-of-vocabulary words for copy mechanism.
"""
def __init__(self, vocab):
self.vocab = vocab
self.word_to_idx = {word: i for i, word in enumerate(vocab)}
self.vocab_size = len(vocab)
def process_source(self, source_tokens):
"""
Convert source tokens to IDs, tracking OOV words.
Returns:
source_ids: List of token IDs (OOV words get extended IDs)
oov_words: List of OOV words encountered
extended_vocab: Original vocab + OOV words
"""
source_ids = []
oov_words = []
oov_to_idx = {}
for token in source_tokens:
if token in self.word_to_idx:
source_ids.append(self.word_to_idx[token])
else:
# OOV word
if token not in oov_to_idx:
oov_to_idx[token] = len(oov_words)
oov_words.append(token)
# Extended ID = vocab_size + oov_index
source_ids.append(self.vocab_size + oov_to_idx[token])
extended_vocab = self.vocab + oov_words
return source_ids, oov_words, extended_vocab
def decode_output(self, output_ids, oov_words):
"""
Convert output IDs back to tokens, using OOV words when needed.
"""
output_tokens = []
for idx in output_ids:
if idx < self.vocab_size:
output_tokens.append(self.vocab[idx])
else:
oov_idx = idx - self.vocab_size
if oov_idx < len(oov_words):
output_tokens.append(oov_words[oov_idx])
else:
output_tokens.append("<unk>")
return output_tokens
# Example
base_vocab = [
"<pad>",
"<unk>",
"<sos>",
"<eos>",
"the",
"said",
"at",
"university",
]
handler = OOVHandler(base_vocab)
source = [
"Dr.",
"Chen",
"said",
"the",
"discovery",
"at",
"Stanford",
"university",
]
source_ids, oov_words, extended_vocab = handler.process_source(source)OOV Handling ================================================== Base vocabulary size: 8 Base vocab: ['<pad>', '<unk>', '<sos>', '<eos>', 'the', 'said', 'at', 'university'] Source: ['Dr.', 'Chen', 'said', 'the', 'discovery', 'at', 'Stanford', 'university'] Processed source IDs: [8, 9, 5, 4, 10, 6, 11, 7] OOV words found: ['Dr.', 'Chen', 'discovery', 'Stanford'] Extended vocabulary: ['<pad>', '<unk>', '<sos>', '<eos>', 'the', 'said', 'at', 'university', 'Dr.', 'Chen', 'discovery', 'Stanford'] Token mapping: Dr. -> 8 (OOV, extended) Chen -> 9 (OOV, extended) said -> 5 (in vocab) the -> 4 (in vocab) discovery -> 10 (OOV, extended) at -> 6 (in vocab) Stanford -> 11 (OOV, extended) university -> 7 (in vocab)
The OOV handler assigns extended vocabulary IDs to words not in the base vocabulary. "Dr.", "Chen", "discovery", and "Stanford" receive IDs starting from the base vocabulary size (8), allowing the copy mechanism to produce these tokens even though they weren't in the original vocabulary. This dynamic vocabulary extension is computed per-input, so different source documents can have different OOV words.
This per-input vocabulary creates a subtle complication at inference time. The model outputs an index into the extended vocabulary. For indices below vocab_size, the index maps to the standard vocabulary as usual. For indices above vocab_size, the model is pointing to an OOV word, and you need to look up which OOV word appeared for this particular input. The decode_output method above handles this mapping, converting extended indices back to the actual OOV strings using the list of OOV words that was built during preprocessing. This means the full pipeline for a single input involves preprocessing to identify OOV words and build extended IDs, a forward pass using those extended IDs, and postprocessing to convert extended IDs back to strings.
During training, we need to handle the case where target tokens are OOV. When a target token is OOV but appears in the source, we want the model to learn to copy it. This requires computing the loss against the extended vocabulary distribution: the model should assign high probability to the OOV token's extended ID, which can only happen through the copy channel. When a target token is OOV and does not appear in the source at all, there is truly nothing the model can do; it must fall back to <unk>, and the loss reflects this irreducible error.
def compute_loss_with_oov(final_dist, target_ids, vocab_size):
"""
Compute cross-entropy loss, handling OOV targets.
If target is OOV but copyable from source, loss is computed
against the extended distribution. If target is truly unknown,
it's mapped to <unk>.
"""
batch_size = target_ids.shape[0]
extended_size = final_dist.shape[1]
# Clamp target IDs to valid range
# OOV targets beyond extended vocab map to <unk> (index 1)
valid_targets = target_ids.clone()
valid_targets[valid_targets >= extended_size] = 1 # <unk>
# Gather probabilities for target tokens
target_probs = final_dist.gather(1, valid_targets.unsqueeze(1)).squeeze(1)
# Negative log likelihood
loss = -torch.log(target_probs + 1e-12)
return loss.mean()
# Example
target_ids = torch.tensor([8, 9, 5]) # "Dr.", "Chen", "said"
# "Dr." and "Chen" are OOV (ids 8, 9), "said" is in vocab (id 5)
# Simulated final distribution (extended vocab)
final_dist = torch.zeros(3, 12) # batch=3, extended_vocab=12
final_dist[0, 8] = 0.7 # High prob for "Dr." (copied)
final_dist[1, 9] = 0.6 # High prob for "Chen" (copied)
final_dist[2, 5] = 0.8 # High prob for "said" (generated)
loss = compute_loss_with_oov(final_dist, target_ids, len(base_vocab))Loss Computation with OOV
==================================================
Target tokens: ['Dr.', 'Chen', 'said']
Target IDs: [8, 9, 5]
Base vocab size: 8
Probabilities assigned to targets:
P('Dr.') = 0.70 (OOV, copied)
P('Chen') = 0.60 (OOV, copied)
P('said') = 0.80 (in vocab, generated)
Loss: 0.3635