Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Copy Mechanism › Mixing Generation and Copying

Michael BrenndoerferUpdated May 19, 20258 min read

Language AI HandbookAll chapters

Part 12 of 67Sequence-to-Sequence

Chapter 89Copy Mechanism › Mixing Generation and Copying

Reading section 5 of 14 · Mixing Generation and Copying · 8 minSection 5 of 14 · 8 min

Section 5 of 14 · 8 min read

Copy Mechanism › Mixing Generation and CopyingOne section · Sections in this chapter5 / 14 ↓

Sections in this chapter14

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

We now have two probability distributions: one over the vocabulary (from the decoder's softmax) and one over input positions (from pointer attention). We also have a switch $p_{\text{gen}}$ that tells us how much to trust each. How do we combine them into a single output distribution?

The naive approach would be to make a hard choice: either generate or copy. But this loses information and creates discontinuities that hurt gradient flow during training. Instead, we use $p_{\text{gen}}$ as a mixing coefficient that smoothly blends both distributions.

Think of the mixing operation as a weighted average of two experts: the generator expert, which knows the full vocabulary and can produce fluent language, and the copier expert, which has direct access to the source tokens regardless of whether they appear in any vocabulary. The parameter $p_{\text{gen}}$ controls how much weight to give each expert. Rather than forcing a discrete commitment, the model holds both possibilities open simultaneously, which is both more numerically stable during training and more flexible at inference.

Consider a word $w$ that we might want to output. There are three cases:

$w$ is only in the vocabulary: It can only be generated, so its probability comes entirely from $P_{\text{vocab}}(w)$ , scaled by $p_{\text{gen}}$ .
$w$ is only in the input (OOV): It can only be copied, so its probability comes from the attention weights on positions containing $w$ , scaled by $(1 - p_{\text{gen}})$ .
$w$ is in both: It receives probability from both sources, which are added together.

This leads to the final probability formula:

P(w) = p_{\text{gen}} \cdot P_{\text{vocab}}(w) + (1 - p_{\text{gen}}) \cdot \sum_{i: x_i = w} a_i

where:

$P(w)$ : the final probability of outputting word $w$
$p_{\text{gen}}$ : the generation probability from the copy switch
$P_{\text{vocab}}(w)$ : the probability assigned to $w$ by the decoder's vocabulary softmax
$a_i$ : the attention weight (pointer probability) for input position $i$
$x_i$ : the token at input position $i$
$\sum_{i: x_i = w}$ : sum over all positions where the input token equals $w$

The summation $\sum_{i: x_i = w}$ deserves attention. If a word appears multiple times in the input, for example "the" might appear at positions 2, 7, and 15, we sum the attention weights from all those positions. This makes intuitive sense: if the model attends to any occurrence of "the" in the input, that attention contributes to the probability of outputting "the."

This summation over all matching positions is a subtle but important design choice. An alternative would be to route attention from each position independently, but that would require knowing in advance which specific occurrence of a repeated word you want to copy. By aggregating over all positions containing the same word, the model sidesteps this ambiguity: it only needs to decide what word to output, not which occurrence to point to.

Let's trace through a concrete example. Suppose we're generating a summary and the input contains "the president Nakamura said." Our vocabulary includes common words but not "Nakamura." At the current step:

The vocabulary distribution assigns: $P_{\text{vocab}}(\text{said}) = 0.1$ , $P_{\text{vocab}}(\text{the}) = 0.15$
The pointer distribution assigns: $a_1 = 0.1$ (the), $a_2 = 0.3$ (president), $a_3 = 0.4$ (Nakamura), $a_4 = 0.2$ (said)
The copy switch outputs: $p_{\text{gen}} = 0.3$ (favoring copying)

For "Nakamura" (OOV, only copyable):

P(\text{Nakamura}) = 0.3 \cdot 0 + 0.7 \cdot 0.4 = 0.28

For "said" (in both vocab and input):

P(\text{said}) = 0.3 \cdot 0.1 + 0.7 \cdot 0.2 = 0.03 + 0.14 = 0.17

The OOV word "Nakamura" receives substantial probability through copying alone, while "said" gets a boost from both sources.

This worked example illustrates a key structural property of the mechanism: words that exist only in the vocabulary rely entirely on $p_{\text{gen}}$ , words that exist only in the source rely entirely on $(1 - p_{\text{gen}})$ , and words that exist in both receive contributions from both channels. The total probability remains valid (sums to 1) because both component distributions are individually normalized and the mixing coefficients $p_{\text{gen}}$ and $(1 - p_{\text{gen}})$ sum to 1. This mathematical cleanliness is one of the reasons the pointer-generator formulation became the standard approach.

In[11]:

Code

def compute_final_distribution(
    p_gen,  # (batch, 1) generation probability
    vocab_dist,  # (batch, vocab_size) vocabulary distribution
    pointer_probs,  # (batch, src_len) attention/pointer distribution
    source_ids,  # (batch, src_len) token IDs in source
    vocab_size,  # int
    oov_ids=None,  # (batch, src_len) IDs for OOV tokens (optional)
):
    """
    Combine generation and copy distributions.

    For words in both vocab and source, probabilities are summed.
    For OOV words (only in source), only copy probability applies.
    """
    batch_size, src_len = source_ids.shape

    # Start with generation distribution, scaled by p_gen
    # Extended vocab includes OOV slots
    if oov_ids is not None:
        max_oov = oov_ids.max().item() + 1
        extended_size = vocab_size + max_oov
    else:
        extended_size = vocab_size

    final_dist = torch.zeros(batch_size, extended_size)
    final_dist[:, :vocab_size] = p_gen * vocab_dist

    # Add copy probabilities
    p_copy = 1 - p_gen  # (batch, 1)

    for b in range(batch_size):
        for i in range(src_len):
            token_id = source_ids[b, i].item()

            # If token is in vocab, add to that position
            if token_id < vocab_size:
                final_dist[b, token_id] += p_copy[b, 0] * pointer_probs[b, i]
            # If OOV, add to extended vocab position
            elif oov_ids is not None:
                oov_idx = oov_ids[b, i].item()
                final_dist[b, vocab_size + oov_idx] += (
                    p_copy[b, 0] * pointer_probs[b, i]
                )

    return final_dist


# Example with a small vocabulary
small_vocab = ["<unk>", "<sos>", "<eos>", "the", "said", "president"]
small_vocab_size = len(small_vocab)

# Source tokens: "the president Nakamura said"
# "Nakamura" is OOV (id = vocab_size, oov_idx = 0)
source_ids = torch.tensor([[3, 5, 6, 4]])  # the, president, <oov>, said
oov_ids = torch.tensor([[0, 0, 0, 0]])  # Only position 2 is OOV

# Simulated distributions
vocab_dist = F.softmax(torch.randn(1, small_vocab_size), dim=-1)
pointer_probs_example = torch.tensor(
    [[0.1, 0.3, 0.4, 0.2]]
)  # High attention on "Nakamura"
p_gen_example = torch.tensor([[0.3]])  # Likely to copy

final_dist = compute_final_distribution(
    p_gen_example,
    vocab_dist,
    pointer_probs_example,
    source_ids,
    small_vocab_size,
    oov_ids,
)

Out[12]:

Console

Final Distribution Computation
==================================================

Source: ['the', 'president', 'Nakamura', 'said']
p_gen = 0.30 (low = prefer copying)

Pointer probabilities:
  the: 0.10
  president: 0.30
  Nakamura: 0.40
  said: 0.20

Final distribution:
  <unk>: 0.023 
  <sos>: 0.020 
  <eos>: 0.148 ████
  the: 0.111 ███
  said: 0.178 █████
  president: 0.240 ███████
  Nakamura (OOV): 0.280 ████████

Out[13]:

Console

With $p_{\text{gen}} = 0.30$:

| Word | In Vocab | In Source | Generation | Copy | **Total** |
|:-----|:--------:|:---------:|-----------:|-----:|----------:|
| the | ✓ | ✓ | 0.041 | 0.070 | **0.111** |
| said | ✓ | ✓ | 0.038 | 0.140 | **0.178** |
| president | ✓ | ✓ | 0.030 | 0.210 | **0.240** |
| Nakamura (OOV) | . | ✓ | 0.000 | 0.280 | **0.280** |

: Final probability distribution combining generation and copy contributions. Words in both vocabulary and source (like "the" and "said") receive probability from both channels. OOV words like "Nakamura" can only be copied, receiving their entire probability from the copy mechanism. {#tbl-probability-combination}

The table reveals the mechanics of probability combination. "Nakamura" receives its entire probability from copying, since it's not in the vocabulary. "the" and "said" appear in both the source and vocabulary, so they receive contributions from both channels. With $p_{\text{gen}} = 0.30$ , the copy channel dominates, which is appropriate when the model needs to preserve specific names from the input.

The output confirms our mathematical analysis. "Nakamura," despite being out-of-vocabulary, receives the highest probability through the copy mechanism. The model can produce this word in its output even though it never appeared in training. Meanwhile, words like "said" and "the" receive probability from both generation and copying, with their contributions weighted by $p_{\text{gen}}$ .

This combination of generation and copying solves the OOV problem while preserving the model's ability to generate fluent, grammatical text. The soft switch learns to route rare words through copying and common words through generation.

It is worth pausing to appreciate what this achieves at training time. The model never receives explicit supervision about which tokens should be copied and which should be generated. It only receives the final target sequence. Yet through backpropagation, it learns to route the gradient signal appropriately: when an OOV target token has high probability only through the copy channel, the gradient flows through the attention weights, encouraging the model to attend more sharply to the relevant source position. When a common token is generated correctly, the gradient flows through the vocabulary softmax. The two pathways train themselves to specialize without any manual annotation of copy versus generate decisions.