Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Attention Masking › Combining Multiple Masks

Michael BrenndoerferUpdated May 27, 20255 min read

Language AI HandbookAll chapters

Part 13 of 67Self-Attention

Chapter 93Attention Masking › Combining Multiple Masks

Reading section 5 of 13 · Combining Multiple Masks · 5 minSection 5 of 13 · 5 min

Section 5 of 13 · 5 min read

Attention Masking › Combining Multiple MasksOne section · Sections in this chapter5 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Real models often need both padding and causal masks simultaneously. A decoder processing batched sequences must handle variable lengths (requiring padding masks) while maintaining autoregressive constraints (requiring causal masks). These two requirements are independent but must be enforced together in every forward pass.

The solution is mathematically elegant: combine masks by addition. If we have a causal mask $M_{\text{causal}}$ and a padding mask $M_{\text{pad}}$ , the combined mask is:

M_{\text{combined}} = M_{\text{causal}} + M_{\text{pad}}

where:

$M_{\text{causal}}$ : the causal mask with 0 for past/current positions and $-\infty$ for future positions
$M_{\text{pad}}$ : the padding mask with 0 for real token positions and $-\infty$ for padding positions
$M_{\text{combined}}$ : the result, which blocks a position if either mask blocks it

Since both masks use $-\infty$ for blocked positions and 0 for allowed positions, the sum gives $-\infty$ wherever either mask blocks attention. A position is only allowed (value 0) when both masks allow it: the position must be in the past (causal constraint) and must be a real token (padding constraint). This is the logical AND of the two constraints, implemented through simple arithmetic addition.

Why does addition work here? Because adding any finite number to $-\infty$ still gives $-\infty$ , and adding zero to any value leaves it unchanged. The mask values act like boolean logic: 0 is "true" (allow) and $-\infty$ is "false" (block), and addition implements the OR operation on the "block" signal.

In[18]:

Code

def combine_masks(*masks):
    """
    Combine multiple attention masks by addition.

    Each mask should use 0 for allowed positions and -inf for blocked.
    The result blocks any position blocked by ANY mask.
    """
    combined = np.zeros_like(masks[0])
    for mask in masks:
        combined = combined + mask
    return combined


# Example: Batch of 2 sequences with different lengths
# Sequence 0: "Hello world" (2 tokens) + 2 padding
# Sequence 1: "The cat sat" (3 tokens) + 1 padding
batch_sequences = [
    ["Hello", "world", "[PAD]", "[PAD]"],
    ["The", "cat", "sat", "[PAD]"],
]
seq_len = 4

# Create padding mask for this batch
padding_mask = create_padding_mask(batch_sequences)
padding_attn_mask = padding_mask_to_attention_mask(padding_mask)

# Create causal mask (same for all sequences)
causal = create_causal_mask(seq_len)
# Expand to batch dimension: (1, seq_len, seq_len) for broadcasting
causal_expanded = causal[np.newaxis, :, :]

# Combine masks
combined_mask = combine_masks(padding_attn_mask, causal_expanded)

Out[19]:

Console

Sequence 0: ['Hello', 'world', '[PAD]', '[PAD]']
Combined mask (0=attend, large negative=block):
[[          0 -1000000000 -2000000000 -2000000000]
 [          0           0 -2000000000 -2000000000]
 [          0           0 -1000000000 -2000000000]
 [          0           0 -1000000000 -1000000000]]

Sequence 1: ['The', 'cat', 'sat', '[PAD]']
Combined mask:
[[          0 -1000000000 -1000000000 -2000000000]
 [          0           0 -1000000000 -2000000000]
 [          0           0           0 -2000000000]
 [          0           0           0 -1000000000]]

For sequence 0 with 2 real tokens, the combined mask blocks:

Upper triangle (causal constraint): future positions are hidden
Columns 2 and 3 (padding positions): padding keys are hidden

The resulting mask allows each position to attend only to past, non-padding positions. This is exactly the correct behavior for a causal language model trained on batched variable-length sequences.

Out[20]:

Visualization

The green cells marked "Y" show where attention is allowed. For sequence 0, only the 2x2 lower-left block is active. For sequence 1, the lower-left triangular 3x3 region allows attention. Both patterns combine causal and padding constraints in a single mask. The visual pattern makes clear how the two masks compose: the causal mask "cuts" along the diagonal, and the padding mask "cuts" along the rightmost columns.