Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Copy Mechanism › Coverage Mechanism

Michael BrenndoerferUpdated May 19, 20258 min read

Language AI HandbookAll chapters

Part 12 of 67Sequence-to-Sequence

Chapter 89Copy Mechanism › Coverage Mechanism

Reading section 10 of 14 · Coverage Mechanism · 8 minSection 10 of 14 · 8 min

Section 10 of 14 · 8 min read

Copy Mechanism › Coverage MechanismOne section · Sections in this chapter10 / 14 ↓

Sections in this chapter14

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

One issue with attention-based models is repetition: the model may attend to the same source positions multiple times, generating repetitive output. The coverage mechanism addresses this by tracking which source positions have already been attended to.

Repetition is a particularly visible failure mode in summarization. Models without coverage mechanisms have a tendency to produce summaries like "The president announced a new plan. The president announced the plan. The plan was announced by the president." This happens because, at each decoding step, the attention mechanism makes decisions based only on the current decoder state and the encoder outputs, with no memory of what it has already attended to. If the decoder state happens to be in a configuration that points attention toward "the president announced," it will generate that phrase again, because nothing in the standard attention formulation prevents it. The coverage mechanism adds exactly this memory: a record of where attention has already been focused.

Coverage Mechanism

Coverage maintains a running sum of attention distributions from all previous decoder steps. This coverage vector is used to penalize re-attending to already-covered positions, reducing repetition in the output.

The coverage vector $c_t$ at decoder step $t$ accumulates all previous attention distributions:

c_t = \sum_{t'=0}^{t-1} a_{t'}

where:

$c_t$ : the coverage vector at step $t$ , with one value per source position
$a_{t'}$ : the attention distribution at previous step $t'$
The sum runs over all previous decoder steps from $0$ to $t-1$

Each element $c_{t,i}$ represents how much total attention has been paid to source position $i$ so far. High values indicate positions that have been heavily attended; low values indicate under-attended positions.

The coverage vector serves two roles. First, it is fed as an additional input to the attention scoring function. This allows the current attention distribution to be conditioned on the coverage history. The attention mechanism learns to lower scores for already-covered positions, effectively discouraging re-attendance through the scoring mechanism itself. Second, a separate coverage loss directly penalizes re-attending to covered positions during training. This provides a stronger gradient signal to reinforce the desired behavior.

This coverage vector is incorporated into the attention computation, encouraging the model to attend to positions with low coverage. Additionally, a coverage loss explicitly penalizes re-attending to already-covered positions during training:

\text{covloss}_t = \sum_i \min(a_{t,i}, c_{t,i})

where:

$\text{covloss}_t$ : the coverage loss at step $t$
$a_{t,i}$ : the current attention weight on source position $i$
$c_{t,i}$ : the accumulated coverage at position $i$
$\min(\cdot, \cdot)$ : takes the element-wise minimum

The intuition behind the $\min$ function: the loss is only incurred when both the current attention $a_{t,i}$ and the past coverage $c_{t,i}$ are high for the same position. If either is low, the contribution to the loss is small. This allows the model to attend to new positions freely while penalizing redundant attention to already-covered content.

the $\min$ formulation is captures more detail than a simple $a_{t,i} \cdot c_{t,i}$ product would be. Consider position $i$ with past coverage $c_{t,i} = 0.9$ and current attention $a_{t,i} = 0.1$ . The product would be $0.09$ , still moderately penalizing the small current attention. But the $\min$ gives $0.1$ . This reflects that the current attention is the binding constraint: since the model is only attending to this position a little, the redundancy penalty is proportionally small. Conversely, if $c_{t,i} = 0.1$ and $a_{t,i} = 0.9$ , you are attending heavily to a position that was barely covered before, which is desirable behavior. The $\min$ gives $0.1$ , a small penalty, which is correct. The $\min$ thus charges a penalty proportional to the smaller of the two quantities, which precisely captures the intuition that clear redundancy requires clear weight in both the current step and the past history.

In[27]:

Code

class CoverageAttention(nn.Module):
    """
    Attention with coverage mechanism to reduce repetition.
    """

    def __init__(self, hidden_dim):
        super().__init__()
        self.hidden_dim = hidden_dim

        # Standard attention parameters
        self.W_encoder = nn.Linear(hidden_dim, hidden_dim, bias=False)
        self.W_decoder = nn.Linear(hidden_dim, hidden_dim, bias=False)
        self.W_coverage = nn.Linear(1, hidden_dim, bias=False)
        self.v = nn.Linear(hidden_dim, 1, bias=False)

    def forward(self, decoder_state, encoder_outputs, coverage, mask=None):
        """
        Compute attention with coverage.

        Args:
            decoder_state: (batch, hidden)
            encoder_outputs: (batch, seq_len, hidden)
            coverage: (batch, seq_len) - sum of previous attention weights
            mask: (batch, seq_len)

        Returns:
            attn_weights: (batch, seq_len)
            coverage_loss: scalar
        """
        batch_size, seq_len, _ = encoder_outputs.shape

        # Project inputs
        encoder_proj = self.W_encoder(encoder_outputs)  # (batch, seq, hidden)
        decoder_proj = self.W_decoder(decoder_state).unsqueeze(
            1
        )  # (batch, 1, hidden)
        coverage_proj = self.W_coverage(
            coverage.unsqueeze(-1)
        )  # (batch, seq, hidden)

        # Compute scores with coverage
        scores = self.v(torch.tanh(encoder_proj + decoder_proj + coverage_proj))
        scores = scores.squeeze(-1)  # (batch, seq_len)

        if mask is not None:
            scores = scores.masked_fill(~mask, float("-inf"))

        attn_weights = F.softmax(scores, dim=-1)

        # Coverage loss: penalize re-attending
        coverage_loss = torch.sum(
            torch.min(attn_weights, coverage), dim=-1
        ).mean()

        return attn_weights, coverage_loss


# Example
# Use dimensions from PointerGeneratorDecoder
cov_hidden_dim = 128
cov_batch_size = 2
cov_src_len = 10

coverage_attn = CoverageAttention(cov_hidden_dim)

# Create matching tensors
cov_decoder_state = torch.randn(cov_batch_size, cov_hidden_dim)
cov_encoder_outputs = torch.randn(cov_batch_size, cov_src_len, cov_hidden_dim)

# Simulated coverage from previous steps
coverage = torch.tensor([[0.0, 0.3, 0.5, 0.1, 0.1, 0.0, 0.0, 0.0, 0.0, 0.0]])
coverage = coverage.expand(cov_batch_size, -1)

attn_weights, cov_loss = coverage_attn(
    cov_decoder_state,
    cov_encoder_outputs,
    coverage,
)

Out[28]:

Console

Coverage Mechanism
==================================================

Previous coverage (positions already attended):
  Position 1: 0.30 ██████
  Position 2: 0.50 ██████████
  Position 3: 0.10 ██
  Position 4: 0.10 ██

New attention weights:
  Position 0: 0.099 █
  Position 1: 0.087 █
  Position 2: 0.101 ██
  Position 3: 0.087 █
  Position 4: 0.101 ██
  Position 5: 0.154 ███
  Position 6: 0.124 ██
  Position 7: 0.088 █
  Position 8: 0.076 █
  Position 9: 0.083 █

Coverage loss: 0.3852
(Lower is better - means less re-attending to covered positions)

Out[29]:

Visualization

Heatmap showing attention weights at each decoding step over source tokens. — Attention distributions at each decoding step. Each row shows where the model attends when generating that step's output. Early steps focus on 'The president' and 'Chen', while later steps shift to 'announced', 'new policy', and the period.

Heatmap showing cumulative coverage accumulation over decoding steps. — Cumulative coverage before each decoding step. High values (darker orange) indicate positions that have received substantial attention in previous steps, discouraging the model from re-attending to them.

The visualization shows how coverage accumulates during generation. In the left panel, each row represents the attention distribution at a single decoding step. The right panel shows the cumulative coverage before each step. Notice how "president" (position 1) quickly accumulates high coverage after step 0, discouraging the model from re-attending to it. By step 4, the coverage mechanism has effectively "used up" the early positions, encouraging the model to attend to later, less-covered positions.

The coverage mechanism illustrates a broader principle in sequence modeling: sometimes the right way to improve a model is not to add more capacity but to add the right inductive bias. A standard LSTM decoder theoretically has enough capacity to remember what it has already generated and avoid repetition, but in practice it does not reliably learn to do so from the training signal alone. The coverage vector explicitly encodes this information in a structured way, making it easy for the attention mechanism to condition on coverage history. This kind of structured auxiliary information often produces better and more sample-efficient learning than hoping the model will discover the same structure implicitly.