Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Attention Masking › Causal Masks

Michael BrenndoerferUpdated May 27, 20258 min read

Language AI HandbookAll chapters

Part 13 of 67Self-Attention

Chapter 93Attention Masking › Causal Masks

Reading section 4 of 13 · Causal Masks · 8 minSection 4 of 13 · 8 min

Section 4 of 13 · 8 min read

Attention Masking › Causal MasksOne section · Sections in this chapter4 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Causal masking, also called look-ahead masking, prevents positions from attending to future positions. This is essential for autoregressive language models that generate text one token at a time. The name "causal" comes from causality: in a causal system, the past influences the present, but the future does not. A causal attention mask enforces the same constraint.

Think of causal masking as the rule that applies when you are doing a fill-in-the-blanks exercise: you can read the words before the blank to figure out what goes there, but you cannot look ahead to the words after the blank. The causal mask enforces this rule systematically across every position in the sequence simultaneously.

During training, we process entire sequences in parallel for efficiency. But the model must learn to predict each position using only past context. Causal masking enforces this constraint: position $i$ can only attend to positions $0, 1, \ldots, i$ , where $i$ is the current position index (0-indexed). This creates an asymmetric attention pattern where earlier positions see less context and later positions see more. Position 0 sees only itself. Position 1 sees itself and position 0. Position $n-1$ sees the entire sequence. This graduated context is exactly what an autoregressive model needs.

Causal Attention

Causal attention restricts each position to attend only to itself and previous positions. This creates a left-to-right information flow where future tokens cannot influence past representations. Causal masking is what allows transformers to train on full sequences in parallel while simulating the sequential left-to-right prediction they perform at inference time.

The practical benefit of causal masking extends beyond preventing cheating. It also allows the transformer to serve as its own training data generator: by masking the sequence causally and computing cross-entropy loss between each position's prediction and the true next token, we get $n$ training signal pairs from a single forward pass on a sequence of length $n$ . This is called "teacher forcing" and it is one of the reasons large language models can train efficiently on massive datasets.

The Causal Mask FormulaLink Copied

How do we formalize "only attend to past positions"? We need a mask that blocks any attention where the key position comes after the query position. Mathematically, the causal mask $M$ is defined as:

M_{ij} = \begin{cases} 0 & \text{if } j \leq i \\ -\infty & \text{if } j > i \end{cases}

where:

$i$ : the query position (row index), representing the token that is currently "attending"
$j$ : the key position (column index), representing the token being attended to
$M_{ij}$ : the mask value added to the attention score at position $(i, j)$ , which is either zero (no effect) or $-\infty$ (complete suppression)

The condition $j \leq i$ means "key position is at or before query position." When this holds, we add 0 (no masking). When $j > i$ , the key is in the future relative to the query, so we add $-\infty$ to block attention completely.

This creates a lower triangular pattern: the diagonal and everything below it are 0, and everything above the diagonal is $-\infty$ . If you visualize the $n \times n$ mask matrix, the allowed region forms a triangle whose tip is at the top-left corner and whose base runs along the bottom row. Row $i$ has exactly $i+1$ allowed positions (positions 0 through $i$ ), growing from 1 at the top to $n$ at the bottom.

Why does this formula make sense? Notice that the lower triangular structure directly encodes the "past only" constraint. The diagonal ( $j = i$ ) represents attending to the current token itself, which is always allowed. Below the diagonal ( $j < i$ ) represents attending to past tokens, also always allowed. Above the diagonal ( $j > i$ ) represents attending to future tokens, which is always blocked. The matrix structure is a direct geometric representation of the "past is visible, future is hidden" rule.

In[13]:

Code

def create_causal_mask(seq_len):
    """
    Create a causal (look-ahead) mask.

    Returns a mask where:
    - 0.0 for positions that CAN be attended (current and past)
    - -inf for positions that CANNOT be attended (future)

    Shape: (seq_len, seq_len)
    """
    # Create lower triangular matrix of ones
    # mask[i, j] = 1 if j <= i (can attend), else 0
    causal = np.tril(np.ones((seq_len, seq_len)))

    # Convert to additive mask
    mask = np.where(causal, 0.0, -1e9)

    return mask


seq_len = 5
causal_mask = create_causal_mask(seq_len)

Out[14]:

Console

Causal mask (0 = can attend, -1e9 = cannot attend):
[[ 0.e+00 -1.e+09 -1.e+09 -1.e+09 -1.e+09]
 [ 0.e+00  0.e+00 -1.e+09 -1.e+09 -1.e+09]
 [ 0.e+00  0.e+00  0.e+00 -1.e+09 -1.e+09]
 [ 0.e+00  0.e+00  0.e+00  0.e+00 -1.e+09]
 [ 0.e+00  0.e+00  0.e+00  0.e+00  0.e+00]]

Position 0 can only attend to itself (only position 0 has value 0 in its row). Position 1 can attend to positions 0 and 1. Position 4 can attend to all positions. The upper triangle is filled with large negative values, blocking all look-ahead. The resulting pattern is exactly the lower triangular structure predicted by the formula.

Visualizing Causal AttentionLink Copied

Let's trace through how causal masking affects attention during sequence processing. We will use a concrete five-token sequence and visualize the resulting attention weight matrix.

In[15]:

Code

# Example sequence
tokens = ["The", "cat", "sat", "on", "mat"]
seq_len = len(tokens)

# Random attention scores (before softmax)
scores = np.random.randn(seq_len, seq_len) * 0.5

# Apply causal mask
causal_mask = create_causal_mask(seq_len)
masked_scores = scores + causal_mask

# Compute attention weights
causal_weights = softmax(masked_scores)

Out[16]:

Visualization

Lower triangular heatmap showing causal attention weights with zeros in upper triangle. — Causal attention pattern for a five-token sequence. The lower triangular structure shows allowed attention, while the upper triangle is completely zeroed out. Position 0 ('The') attends only to itself, while position 4 ('mat') distributes attention across all five tokens based on relevance.

The triangular structure is clear. "The" at position 0 attends entirely to itself (weight 1.00) because it has no past context. "cat" can attend to "The" or itself, distributing its attention between the two based on similarity scores. By position 4, "mat" distributes attention across all five positions based on learned relevance patterns. This graduated context mirrors the information available to an autoregressive model during generation: as more of the sequence has been generated, more context is available.

This structure ensures that when training on a sequence like "The cat sat on mat," each position learns to predict the next token using only information from positions to its left. The training procedure and the inference procedure are now perfectly aligned, which is why causal masking is sometimes described as making training and inference "consistent" with each other.

Why Causal Masking Enables Parallel TrainingLink Copied

Without causal masking, training autoregressive models would be painfully slow. We would need to generate one token at a time, feeding each output back as input for the next step. On a sequence of length 512, this means 512 sequential forward passes. Causal masking eliminates this bottleneck entirely.

In[17]:

Code

# During training, we have the full sequence
training_sequence = ["<s>", "The", "cat", "sat", "</s>"]
target_sequence = ["The", "cat", "sat", "</s>", "<pad>"]

# The model predicts each position simultaneously
# Causal masking ensures each prediction only sees past context

# Position 0: sees only "<s>" -> predicts "The"
# Position 1: sees "<s> The" -> predicts "cat"
# Position 2: sees "<s> The cat" -> predicts "sat"
# Position 3: sees "<s> The cat sat" -> predicts "</s>"

# All predictions happen in one forward pass!

Each row of the attention matrix corresponds to one prediction task. Row 0 uses context $[x_0]$ , row 1 uses context $[x_0, x_1]$ , and so on. The causal mask automatically provides the correct context for each prediction position without needing separate forward passes. A single matrix multiplication and one softmax computation delivers all $n$ predictions simultaneously. This parallelism is what makes training large language models on trillions of tokens computationally feasible.

The efficiency gain is not merely constant-factor. Without causal masking, training on sequences of length $n$ requires $n$ sequential steps, each involving a full forward pass. With causal masking, the same work reduces to a single forward pass. On modern hardware where parallelism is the primary source of speed, this is the difference between a technique that is practical and one that is not. GPT-2, GPT-3, and virtually every other large autoregressive language model owes its training efficiency to this one masking technique.