Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Scaled Dot-Product Attention › Masking for Causal Attention

Michael BrenndoerferMay 25, 20257 min read

Language AI HandbookAll chapters

Part 13 of 67Self-Attention

Chapter 92Scaled Dot-Product Attention › Masking for Causal Attention

Reading section 7 of 15 · Masking for Causal Attention · 7 minSection 7 of 15 · 7 min

Section 7 of 15 · 7 min read

Scaled Dot-Product Attention › Masking for Causal AttentionOne section · Sections in this chapter7 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

So far, attention has been bidirectional: each position can attend to every other position, including those that come later in the sequence. This is fine for encoding tasks like classification or feature extraction, where the full sequence is available at inference time. But for language generation, where we predict token $t+1$ given tokens $1$ through $t$ , we need to prevent each position from seeing future tokens. This is causal (or autoregressive) attention.

The reason this constraint is critical goes beyond preventing "cheating." If a model is allowed to attend to future tokens during training, it learns to predict tokens by copying from the future rather than modeling language. The training loss would be low, but the model would be useless for generation, where the future tokens are exactly what we're trying to predict. Causal masking enforces the correct inductive bias: the model must learn to predict each token from the past context alone.

Causal Attention

Causal attention (also called masked self-attention) restricts each position to attend only to itself and earlier positions. This preserves the autoregressive property needed for text generation: each token's representation depends only on the tokens that preceded it. The GPT family of models uses causal attention exclusively. Encoder-only models like BERT use bidirectional attention, relying on the full context in both directions. Encoder-decoder models like T5 use causal attention in the decoder and bidirectional attention in the encoder.

The mechanism is straightforward: before applying softmax, we add a very large negative value (effectively $-\infty$ ) to the scores for all positions that should be masked. After softmax, $e^{-\infty} = 0$ , so those positions receive zero attention weight. The elegance of this approach is that it requires no changes to the softmax function itself: the masking is applied to the inputs of softmax, and softmax naturally produces zero weights for $-\infty$ inputs.

The causal mask is an upper-triangular matrix of True values (above the diagonal represents future positions):

M_{ij} = \begin{cases} \text{True (mask)} & \text{if } j > i \\ \text{False (keep)} & \text{if } j \leq i \end{cases}

where $i$ is the query position (row) and $j$ is the key position (column). Position $i$ can attend to positions $0, 1, \ldots, i$ but not to positions $i+1, i+2, \ldots, n-1$ . The diagonal entries ( $j = i$ , self-attention) are not masked: each token can always attend to itself.

In[17]:

Code

def make_causal_mask(n):
    """Create an upper-triangular causal mask.
    True means 'mask out this position'.
    """
    mask = np.triu(np.ones((n, n), dtype=bool), k=1)
    return mask


# Create causal mask for sequence of length n
causal_mask = make_causal_mask(n)

Out[18]:

Console

Causal mask (True = masked out):
[[0 1 1 1 1]
 [0 0 1 1 1]
 [0 0 0 1 1]
 [0 0 0 0 1]
 [0 0 0 0 0]]

Meaning: row i can only attend to columns 0..i (its own and previous positions)

In[19]:

Code

# Apply causal masking
output_causal, weights_causal = scaled_dot_product_attention(
    Q, K, V, mask=causal_mask
)

Out[20]:

Console

Causal attention weights (upper triangle is zero):
[[1.     0.     0.     0.     0.    ]
 [0.5533 0.4467 0.     0.     0.    ]
 [0.4505 0.2741 0.2755 0.     0.    ]
 [0.2376 0.3166 0.2666 0.1792 0.    ]
 [0.1575 0.2351 0.2459 0.2124 0.1491]]

Row sums still equal 1 (weights redistribute among unmasked positions):
[1. 1. 1. 1. 1.]

With causal masking, token 0 attends only to itself, token 1 attends to tokens 0 and 1, and so on. The weight in each row still sums to 1, since softmax normalizes over only the unmasked positions. This is exactly what language models like GPT use during training: each token in the sequence produces predictions for the next token, with the mask ensuring the model never peeks ahead. An important practical benefit is that this allows processing the entire sequence in parallel during training: rather than processing token 1, then token 2, then token 3, we can process all tokens simultaneously with the causal mask ensuring each position only sees its allowed context. This is a major advantage over recurrent neural networks, which are inherently sequential.

Out[21]:

Visualization

Binary heatmap of causal attention mask showing lower-triangular structure. — Causal attention mask for a 5-token sequence. White cells are allowed (query can attend to key), dark cells are blocked. The lower-triangular structure ensures each position only attends to earlier positions and itself, enforcing the autoregressive property needed for text generation. This allows the full sequence to be processed in parallel during training.

Attention weight heatmap with upper triangle zeroed out by causal masking. — Attention weights after applying the causal mask. The upper triangle is exactly zero, confirming no future token contributes to any current token's representation. Weights within the allowed lower triangle sum to 1 for each row, with the redistribution becoming more concentrated for early tokens that have fewer allowed positions.

Padding MasksLink Copied

Beyond causal masking, transformers also use padding masks to handle variable-length sequences. When batching sequences of different lengths, shorter sequences are padded with a special token to reach a common length. The model should not attend to padding positions, since they contain no real information: a padded position is an artifact of the batching process, not a meaningful part of the input.

The problem with attending to padding positions is subtle but real. If a padding token happens to receive a high dot-product score, the softmax will assign it a non-trivial weight, and the padding token's value vector will contribute to the output. Since padding tokens carry no semantic content, this is wasted attention capacity at best and harmful noise at worst. The padding mask removes this possibility by zeroing out attention weights for all padding positions.

A padding mask marks padded positions as invalid by setting the corresponding columns of the score matrix to $-\infty$ before softmax, just as the causal mask does for future positions. Combined with the causal mask (when needed), the model correctly ignores both future tokens and padding tokens.

In[22]:

Code

def make_padding_mask(n, valid_length):
    """Create a padding mask.
    Positions >= valid_length are masked as padding.

    Returns a (1, n) mask that broadcasts to (n, n) when combined with scores.
    """
    # Column j is masked if j >= valid_length
    mask = np.arange(n) >= valid_length
    # Broadcast to (n, n) by repeating for all queries
    return np.broadcast_to(mask[np.newaxis, :], (n, n))


# Simulate sequence where only first 3 of 5 tokens are real
valid_length = 3
padding_mask = make_padding_mask(n, valid_length)

Out[23]:

Console

Padding mask (True = padded, should be masked):
[[0 0 0 1 1]
 [0 0 0 1 1]
 [0 0 0 1 1]
 [0 0 0 1 1]
 [0 0 0 1 1]]

In[24]:

Code

# Combine causal and padding masks
combined_mask = causal_mask | padding_mask

# Apply combined mask
output_masked, weights_masked = scaled_dot_product_attention(
    Q, K, V, mask=combined_mask
)

Out[25]:

Console

Combined (causal + padding) mask:
[[0 1 1 1 1]
 [0 0 1 1 1]
 [0 0 0 1 1]
 [0 0 0 1 1]
 [0 0 0 1 1]]

Attention weights with combined mask:
[[1.     0.     0.     0.     0.    ]
 [0.5533 0.4467 0.     0.     0.    ]
 [0.4505 0.2741 0.2755 0.     0.    ]
 [0.2895 0.3858 0.3247 0.     0.    ]
 [0.2467 0.3682 0.3851 0.     0.    ]]

The first three tokens attend only to earlier (non-padded) tokens within their causal window. Tokens 3 and 4 are padding positions; in practice we discard their outputs, but their attention patterns correctly reflect only valid positions. Notice that combining masks is simple: we take the logical OR of the causal mask and the padding mask, and any position that is masked by either constraint is masked in the combined mask.