Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Cross-Attention › Cross-Attention Masking

Michael BrenndoerferUpdated June 18, 20255 min read

Language AI HandbookAll chapters

Part 16 of 67Transformer Architectures

Chapter 114Cross-Attention › Cross-Attention Masking

Reading section 5 of 14 · Cross-Attention Masking · 5 minSection 5 of 14 · 5 min

Section 5 of 14 · 5 min read

Cross-Attention › Cross-Attention MaskingOne section · Sections in this chapter5 / 14 ↓

Sections in this chapter14

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Unlike causal self-attention in decoders, cross-attention typically does not require causal masking. The decoder can attend to any position in the encoder output because the encoder sequence is fully processed before decoding begins. There's no "future information" in the encoder to hide. The encoder has already seen the entire source sentence, so its output at every position already incorporates global context. When the decoder queries the encoder, it can freely look at any position, including positions corresponding to the end of the source sentence, without any information leak.

This contrasts sharply with decoder self-attention, where causal masking is essential. In decoder self-attention, positions can only attend to earlier positions because the decoder is generating tokens sequentially: if it could attend to future tokens, it would be "cheating" by seeing answers before they are generated. Cross-attention has no such concern because the encoder output is not being generated; it is a fixed input representation.

However, cross-attention does require padding masking when processing batches with variable-length source sequences. Real-world datasets contain sentences of many different lengths. If you want to process a batch of several sentences simultaneously, you need to pad the shorter sequences to match the length of the longest sequence in the batch. These padding tokens carry no meaningful information, and the decoder should not attend to them. If it did, the padding would inject noise into the decoder's representations, causing the model to incorporate garbage information when computing its attention-weighted average of the encoder.

The solution is to set the attention scores for padding positions to a very large negative number (like $-10^9$ ) before applying softmax. Since $e^{-10^9} \approx 0$ , these positions receive neededly zero weight after softmax, effectively masking them out. The probability mass that would have gone to padding positions instead concentrates on the valid encoder positions. This ensures the decoder only attends to meaningful content.

In[12]:

Code

def cross_attention_with_mask(Q, K, V, mask=None):
    """
    Cross-attention with optional encoder padding mask.

    Args:
        Q: Queries from decoder (n_dec, d_k)
        K: Keys from encoder (n_enc, d_k)
        V: Values from encoder (n_enc, d_v)
        mask: Boolean mask (n_enc,) where True = valid, False = padding

    Returns:
        output: Attended representations (n_dec, d_v)
        weights: Attention weights (n_dec, n_enc)
    """
    d_k = Q.shape[-1]
    scores = Q @ K.T / np.sqrt(d_k)

    # Apply padding mask
    if mask is not None:
        # Expand mask for broadcasting: (1, n_enc)
        mask = mask.reshape(1, -1)
        # Set padded positions to large negative value
        scores = np.where(mask, scores, -1e9)

    attention_weights = softmax(scores)
    output = attention_weights @ V

    return output, attention_weights

Let's see how masking affects attention:

In[13]:

Code

# Simulate a padded encoder sequence
# Original: "The cat sat" (3 tokens), padded to length 6
encoder_mask = np.array([True, True, True, False, False, False])

# Compute masked cross-attention
output_masked, weights_masked = cross_attention_with_mask(
    Q, K, V, mask=encoder_mask
)

Out[14]:

Console

Attention weights with padding mask:
(Columns 3-5 are padding, should have zero weight)

Weights shape: (4, 6)

Weight matrix (rounded):
[[0.228 0.269 0.503 0.    0.    0.   ]
 [0.27  0.386 0.344 0.    0.    0.   ]
 [0.323 0.431 0.246 0.    0.    0.   ]
 [0.439 0.39  0.171 0.    0.    0.   ]]

Row sums: [1. 1. 1. 1.]
Columns 3-5 sum: 0.0

The masking ensures that padded positions receive zero attention weight. The softmax is applied only over valid encoder positions (columns 0-2), and the probability mass distributes across those positions. This prevents the decoder from incorporating garbage information from padding tokens, which is especially important for shorter sequences in a batch that have many padding tokens appended.

In practice, the implementation of padding masks is straightforward but must be careful about broadcasting. The mask has shape $(n_{\text{enc}},)$ , but scores have shape $(n_{\text{dec}} \times n_{\text{enc}})$ , so the mask needs to be broadcast across the decoder dimension. Reshaping the mask to $(1, n_{\text{enc}})$ allows NumPy (or PyTorch) to broadcast it correctly, applying the same mask to all decoder positions simultaneously.

Out[15]:

Visualization

Heatmap showing attention weights spread across all 6 encoder positions. — Without masking, attention distributes across all 6 positions, including padding tokens that contain garbage information. The three padding positions on the right receive spurious attention weight.

Heatmap showing attention weights concentrated on first 3 positions with zeros for padded positions. — With masking, attention concentrates entirely on valid tokens. Padded positions receive exactly zero weight, and the probability mass redistributes to the three valid positions.

The comparison makes the effect of masking clear. Without masking, attention bleeds into padding positions, corrupting the decoder's representations with meaningless information. With masking, all attention concentrates on the valid tokens, and the model ignores padding entirely. The right-hand heatmap shows darker cells in the first three columns (valid positions) and empty cells in the last three columns (padding), which is exactly the behavior we want.