Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Copy Mechanism › Computing the Copy Probability

Michael BrenndoerferUpdated May 19, 20255 min read

Language AI HandbookAll chapters

Part 12 of 67Sequence-to-Sequence

Chapter 89Copy Mechanism › Computing the Copy Probability

Reading section 4 of 14 · Computing the Copy Probability · 5 minSection 4 of 14 · 5 min

Section 4 of 14 · 5 min read

Copy Mechanism › Computing the Copy ProbabilityOne section · Sections in this chapter4 / 14 ↓

Sections in this chapter14

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

A pure pointer network can only copy from the input, but real text generation requires both copying and generating. Consider summarizing "Dr. Chen announced the results." We want to copy "Dr. Chen" (a rare name) but generate common words like "announced" and "the" from our vocabulary. How does the model decide which strategy to use at each step?

The solution introduces a soft switch between two modes: generating from vocabulary versus copying from input. Rather than making a hard binary choice, we compute a generation probability $p_{\text{gen}} \in (0, 1)$ that smoothly interpolates between the two strategies. When $p_{\text{gen}}$ is high, the model favors generation; when low, it favors copying.

The key insight is that a hard binary choice would be problematic for several reasons. First, the correct strategy is not always obvious from the current word alone. For a word like "bank," whether to copy it from the source or generate it from vocabulary depends on whether the source document is about finance or rivers, which is a semantic question that requires context. Second, hard decisions create discontinuities in the computation graph that make gradient-based training difficult. The soft switch produces smooth gradients throughout, allowing the model to learn the copy-versus-generate decision jointly with all other parameters.

What information should determine this switch? Intuitively, the decision depends on:

The context vector $c_t$ : What part of the input is the model attending to? If attention focuses on a rare word, copying makes sense.
The decoder state $s_t$ : What has the model generated so far? This captures the "momentum" of the generation process.
The previous token $x_t$ : What did we just output? After generating "Dr.", we likely want to copy the following name.

We combine these three signals through a linear combination, then squash the result to $(0, 1)$ using the sigmoid function:

p_{\text{gen}} = \sigma(w_c^T c_t + w_s^T s_t + w_x^T x_t + b)

where:

$p_{\text{gen}}$ : the probability of generating from the vocabulary (vs. copying from input)
$\sigma(\cdot)$ : the sigmoid function, $\sigma(z) = 1/(1 + e^{-z})$ , which maps any real number to the range $(0, 1)$
$c_t$ : the context vector from attention at decoder step $t$
$s_t$ : the decoder hidden state at step $t$
$x_t$ : the decoder input embedding at step $t$ (typically the previous output token)
$w_c, w_s, w_x$ : learnable weight vectors that determine how much each input contributes to the decision
$b$ : a learnable bias term that shifts the default behavior toward generating or copying

The model learns the weights $w_c, w_s, w_x$ during training. It discovers patterns like "when the context vector indicates a rare word and the decoder state suggests we're in the middle of a named entity, favor copying." These patterns emerge from the data without explicit programming.

The choice of three input signals reflects thoughtful design. The context vector $c_t$ tells the model what part of the source it is currently attending to: if attention is concentrated on a rare proper noun, copying is likely appropriate. The decoder state $s_t$ captures the sequential context of what has been generated: after producing "Dr." the model should understand it is likely in a name sequence and prepare to copy the following word. The previous token embedding $x_t$ provides the most immediate signal: the specific word just generated often strongly implies what comes next. Together these three sources of information give the switch enough context to make good decisions consistently.

In[9]:

Code

class CopySwitch(nn.Module):
    """
    Computes the probability of generating vs copying.
    """

    def __init__(self, hidden_dim, embed_dim):
        super().__init__()
        # Linear combination for p_gen
        self.w_context = nn.Linear(hidden_dim, 1, bias=False)
        self.w_state = nn.Linear(hidden_dim, 1, bias=False)
        self.w_input = nn.Linear(embed_dim, 1, bias=False)
        self.bias = nn.Parameter(torch.zeros(1))

    def forward(self, context, decoder_state, decoder_input):
        """
        Compute generation probability.

        Args:
            context: (batch, hidden_dim) - attention context vector
            decoder_state: (batch, hidden_dim) - decoder hidden state
            decoder_input: (batch, embed_dim) - current input embedding

        Returns:
            p_gen: (batch, 1) - probability of generating from vocabulary
        """
        score = (
            self.w_context(context)
            + self.w_state(decoder_state)
            + self.w_input(decoder_input)
            + self.bias
        )
        p_gen = torch.sigmoid(score)
        return p_gen


# Example
embed_dim = 32
copy_switch = CopySwitch(hidden_dim, embed_dim)

context = torch.randn(batch_size, hidden_dim)
decoder_input = torch.randn(batch_size, embed_dim)

p_gen = copy_switch(context, decoder_state, decoder_input)

Out[10]:

Console

Copy Switch Output
==================================================

Generation probability p_gen:
  Batch 0: p_gen = 0.512
           p_copy = 0.488
  Batch 1: p_gen = 0.293
           p_copy = 0.707

The output reveals how the copy switch behaves: $p_{\text{gen}}$ values near 0 indicate the model prefers copying, while values near 1 indicate generation. In practice, the model learns to produce low $p_{\text{gen}}$ for rare words and names, and high $p_{\text{gen}}$ for common function words.

Empirically, well-trained pointer-generator models develop strong separation in $p_{\text{gen}}$ values across token types. Articles, prepositions, and common verbs like "the," "of," and "said" typically receive $p_{\text{gen}}$ above 0.8. Named entities and numbers typically receive $p_{\text{gen}}$ below 0.2. The interesting cases are mid-frequency words that could reasonably come from either source, where the model must use context to decide. This learned specialization is not hand-coded: it emerges entirely from exposure to (source, summary) pairs during training.