Attention Masking: Controlling Information Flow

Self-attention computes pairwise interactions between all positions in a sequence, letting every token gather information from every other token. This generality is one of attention's greatest strengths, but it also raises an immediate practical problem: sometimes we need to block certain interactions entirely. A model generating text one token at a time must not be allowed to peek at the tokens that come after the current position. Sequences padded to a uniform length so they can be batched together should not let real tokens attend to meaningless padding. Controlling which positions are "visible" to which others is not a minor detail but a foundational design decision that shapes what a transformer can and cannot do.

Think of attention masking as a privacy screen on a monitor. Without the screen, anyone walking past can read what is displayed. With the screen, only people sitting in the correct position see the content, while everyone else sees darkness. In the same way, an attention mask designates which query-key pairs are "in view" (allowed to interact) and which are "blocked" (prevented from interacting). The screen does not change the underlying pixels; it simply controls visibility. The mask does not change the attention mechanism; it simply controls which scores contribute to the final weighted sum.

Masking modifies the attention computation by adding large negative values to specific positions before the softmax. Recall that attention weights are computed as:

where:

• $\alpha_{ij}$: the attention weight from query position $i$ to key position $j$, representing how much position $i$ "attends to" position $j$
• $s_{ij}$: the raw attention score between positions $i$ and $j$, computed as the dot product of their query and key vectors (scaled by $1/\sqrt{d_k}$)
• $n$: the sequence length

When we add a large negative value (like $-10^9$) to a score $s_{ij}$, the exponential $\exp(s_{ij} - 10^9)$ becomes vanishingly small, driving $\alpha_{ij}$ to near-zero. These masked positions effectively disappear from the weighted sum. This simple mechanism enables causal language modeling, efficient batch processing, and custom attention patterns.

The elegance of this approach is that it requires no changes to the attention mechanism itself. The formula remains identical. The mask is simply an extra additive term inserted at one specific point in the computation, yet that term determines which positions survive softmax. Masked positions contribute nothing to the output, so from the model's perspective, they simply do not exist during that forward pass.

Understanding attention masking deeply matters for several reasons. First, it is the difference between a model that cheats during training (by reading future tokens) and one that learns to predict. Second, it is what makes batched training of variable-length sequences possible without corrupting representations. Third, it is the foundation of efficient attention variants that scale transformers to sequences far longer than standard full attention can handle. Every serious transformer practitioner needs to understand that masks exist, how they work, when to apply them, and what happens when they are misconfigured.

Before building the machinery of masking, it is worth pausing to understand why the problem exists in the first place. The need for masking arises from a fundamental tension in how transformers are trained versus how they are used at inference time.

Consider training a language model to predict the next word. Given the sentence "The cat sat on the mat," the model should learn to predict "cat" given "The," "sat" given "The cat," "on" given "The cat sat," and so on. This sequential prediction task is called autoregressive modeling because each prediction is conditioned on all prior predictions. The natural way to train such a model would be to feed it one token at a time, compute a prediction, then move on to the next position. But this sequential approach is tragically slow: you can only process one token at a time, so training on millions of sentences would take an impractical amount of time.

The solution that transformers use is to process the entire sequence in parallel during training, computing predictions for all positions simultaneously. This is possible because at training time we already have the complete target sequence. We can process "The cat sat on the mat" all at once, compute predictions at every position, and compare each prediction to the correct next token. But this creates the cheating problem: standard self-attention allows position 1 ("cat") to see position 3 ("on"), which means the model can trivially predict "sat" by simply reading ahead to see what comes next. The learned "predictions" would be meaningless because they are not really predictions at all.

Masking solves this by blocking attention to future positions during training. The model learns to predict each token using only past context, exactly matching the autoregressive setup it will encounter during generation at inference time. The causal mask is thus a training-time simulation of the inference-time constraint: at inference, future tokens do not exist yet, and the mask enforces the same condition during training even though the tokens are already there.

Padding presents a different but equally important challenge. When batching sequences of different lengths, we pad shorter sequences to match the longest. But these padding tokens carry no meaning whatsoever. They are placeholders inserted purely for computational convenience. Allowing real tokens to attend to padding would corrupt their representations by incorporating noise into the weighted sum. A model that learns to interpret padding as meaningful signal would behave erratically on sequences of different lengths. Masking removes padding from the attention computation entirely. This ensures real tokens see only other real tokens.

The key insight is that both masking problems, causal masking and padding masking, have the same solution: add a large negative value to the scores of positions that should be invisible. The mechanism is identical; only the pattern of masked positions differs. This unification is not accidental. It reflects the deeper structure of the attention computation, where everything flows through a single softmax normalization that is highly sensitive to the relative magnitudes of its inputs.

Real-world text comes in varying lengths. "Hello" has one token while "The quick brown fox jumps over the lazy dog" has nine. To process multiple sequences efficiently in a batch, we pad shorter sequences to a common length using a special padding token. Think of padding as blank lines at the bottom of a form: they are there so every form has the same number of lines, but they carry no information and should be ignored when reading the content.

The practical consequence of padding is that every batch contains some positions that should not influence any computation. If we naively apply self-attention without masking, those padding positions will accumulate information from real tokens, and real tokens will waste attention capacity attending to padding positions. Neither interaction is meaningful. A token like "[PAD]" has no semantic content, so attending to it pollutes the output representation with noise.

The severity of this problem scales with the degree of padding in the batch. In a batch where most sequences are short but one is very long, most positions across most sequences are padding. In that scenario, a large fraction of every attention matrix would be wasted on padding interactions. The learned representations would be noisy, training would be less efficient, and the model might learn spurious patterns based on the position of padding tokens rather than the content of real tokens.

The padding mask is a boolean array where True marks real tokens and False marks padding. To apply this mask to attention scores, we need to convert it into an additive mask. This conversion is the bridge between the "which positions are real" question and the "how much should each position contribute" answer.

We have a boolean padding mask, but attention needs numerical scores. How do we bridge this gap? The answer lies in understanding how softmax behaves with extreme values.

Softmax converts a vector of arbitrary real numbers into a probability distribution. Every element becomes positive, and all elements sum to one. The function achieves this through exponentiation followed by normalization. The key insight is that softmax is sensitive to relative differences between values, not their absolute magnitudes. When computing softmax, each score is exponentiated, then divided by the sum of all exponentials:

where:

• $z_i$: the $i$-th element of the input vector, which is the attention score for one query-key pair
• $e^{z_i}$: the exponential of that score, which converts any real number to a positive value
• $\sum_j e^{z_j}$: the sum of exponentials across all positions, which is the normalizing constant ensuring the outputs sum to one

If one score is extremely negative while others are moderate, its exponential becomes vanishingly small. To illustrate, consider applying softmax to a vector with one large negative value:

The third element receives essentially zero weight. This happens because $e^{-10000}$ is astronomically small compared to $e^{2.0}$ and $e^{1.5}$. In practice, $e^{-10000}$ is so close to zero that floating-point arithmetic rounds it to exactly zero, so the masked position truly disappears from the computation.

Why does this formula make sense? Notice that the exponential function is monotone: larger inputs produce larger outputs. By making one input dramatically smaller than all the others, we guarantee its contribution to the normalization sum is negligible, and its share of the normalized output is negligible too. We are essentially creating a value so small that, in the competition of exponentials, it loses completely.

This gives us our masking strategy: add a large negative value to positions we want to block. We typically use $-10^9$ rather than true infinity to avoid numerical edge cases, though both work in practice. True negative infinity can cause NaN values in edge cases where entire rows are masked (since $0/0$ becomes NaN), whereas $-10^9$ keeps the arithmetic well-defined.

The attention mask has shape (batch, seq_len, seq_len). For sequence 0, the last column contains large negative values because position 3 is padding. Every query position will receive near-zero attention weight for position 3 after softmax. Notice that we mask along the key dimension, not the query dimension. The reason is that we want to prevent any query from attending to a padding key, but we do not need to prevent a padding position from being a query: in practice the output at padding positions is simply discarded, so it does not matter what the padding query "attends to."

Let's see how padding masks affect attention weights in practice. We'll create random attention scores and compare the distributions with and without masking.

The contrast is stark. Without masking, attention flows to all positions including padding. The padding tokens absorb 10-30% of the attention weight at each query position. With the mask applied, padding positions receive exactly 0.00 weight, and the remaining attention redistributes entirely to real tokens. This ensures padding never contaminates token representations.

The redistribution effect is particularly important: the two real tokens now share 100% of the attention weight instead of roughly 50-70%. The model has more focused, meaningful signal to work with. Over millions of training examples, this difference accumulates into substantially cleaner representations.

Causal masking, also called look-ahead masking, prevents positions from attending to future positions. This is essential for autoregressive language models that generate text one token at a time. The name "causal" comes from causality: in a causal system, the past influences the present, but the future does not. A causal attention mask enforces the same constraint.

Think of causal masking as the rule that applies when you are doing a fill-in-the-blanks exercise: you can read the words before the blank to figure out what goes there, but you cannot look ahead to the words after the blank. The causal mask enforces this rule systematically across every position in the sequence simultaneously.

During training, we process entire sequences in parallel for efficiency. But the model must learn to predict each position using only past context. Causal masking enforces this constraint: position $i$ can only attend to positions $0, 1, \ldots, i$, where $i$ is the current position index (0-indexed). This creates an asymmetric attention pattern where earlier positions see less context and later positions see more. Position 0 sees only itself. Position 1 sees itself and position 0. Position $n-1$ sees the entire sequence. This graduated context is exactly what an autoregressive model needs.

The practical benefit of causal masking extends beyond preventing cheating. It also allows the transformer to serve as its own training data generator: by masking the sequence causally and computing cross-entropy loss between each position's prediction and the true next token, we get $n$ training signal pairs from a single forward pass on a sequence of length $n$. This is called "teacher forcing" and it is one of the reasons large language models can train efficiently on massive datasets.

How do we formalize "only attend to past positions"? We need a mask that blocks any attention where the key position comes after the query position. Mathematically, the causal mask $M$ is defined as:

where:

• $i$: the query position (row index), representing the token that is currently "attending"
• $j$: the key position (column index), representing the token being attended to
• $M_{ij}$: the mask value added to the attention score at position $(i, j)$, which is either zero (no effect) or $-\infty$ (complete suppression)

The condition $j \leq i$ means "key position is at or before query position." When this holds, we add 0 (no masking). When $j > i$, the key is in the future relative to the query, so we add $-\infty$ to block attention completely.

This creates a lower triangular pattern: the diagonal and everything below it are 0, and everything above the diagonal is $-\infty$. If you visualize the $n \times n$ mask matrix, the allowed region forms a triangle whose tip is at the top-left corner and whose base runs along the bottom row. Row $i$ has exactly $i+1$ allowed positions (positions 0 through $i$), growing from 1 at the top to $n$ at the bottom.

Why does this formula make sense? Notice that the lower triangular structure directly encodes the "past only" constraint. The diagonal ($j = i$) represents attending to the current token itself, which is always allowed. Below the diagonal ($j < i$) represents attending to past tokens, also always allowed. Above the diagonal ($j > i$) represents attending to future tokens, which is always blocked. The matrix structure is a direct geometric representation of the "past is visible, future is hidden" rule.

Position 0 can only attend to itself (only position 0 has value 0 in its row). Position 1 can attend to positions 0 and 1. Position 4 can attend to all positions. The upper triangle is filled with large negative values, blocking all look-ahead. The resulting pattern is exactly the lower triangular structure predicted by the formula.

Let's trace through how causal masking affects attention during sequence processing. We will use a concrete five-token sequence and visualize the resulting attention weight matrix.

The triangular structure is clear. "The" at position 0 attends entirely to itself (weight 1.00) because it has no past context. "cat" can attend to "The" or itself, distributing its attention between the two based on similarity scores. By position 4, "mat" distributes attention across all five positions based on learned relevance patterns. This graduated context mirrors the information available to an autoregressive model during generation: as more of the sequence has been generated, more context is available.

This structure ensures that when training on a sequence like "The cat sat on mat," each position learns to predict the next token using only information from positions to its left. The training procedure and the inference procedure are now perfectly aligned, which is why causal masking is sometimes described as making training and inference "consistent" with each other.

Without causal masking, training autoregressive models would be painfully slow. We would need to generate one token at a time, feeding each output back as input for the next step. On a sequence of length 512, this means 512 sequential forward passes. Causal masking eliminates this bottleneck entirely.

Each row of the attention matrix corresponds to one prediction task. Row 0 uses context $[x_0]$, row 1 uses context $[x_0, x_1]$, and so on. The causal mask automatically provides the correct context for each prediction position without needing separate forward passes. A single matrix multiplication and one softmax computation delivers all $n$ predictions simultaneously. This parallelism is what makes training large language models on trillions of tokens computationally feasible.

The efficiency gain is not merely constant-factor. Without causal masking, training on sequences of length $n$ requires $n$ sequential steps, each involving a full forward pass. With causal masking, the same work reduces to a single forward pass. On modern hardware where parallelism is the primary source of speed, this is the difference between a technique that is practical and one that is not. GPT-2, GPT-3, and virtually every other large autoregressive language model owes its training efficiency to this one masking technique.

Real models often need both padding and causal masks simultaneously. A decoder processing batched sequences must handle variable lengths (requiring padding masks) while maintaining autoregressive constraints (requiring causal masks). These two requirements are independent but must be enforced together in every forward pass.

The solution is mathematically elegant: combine masks by addition. If we have a causal mask $M_{\text{causal}}$ and a padding mask $M_{\text{pad}}$, the combined mask is:

where:

• $M_{\text{causal}}$: the causal mask with 0 for past/current positions and $-\infty$ for future positions
• $M_{\text{pad}}$: the padding mask with 0 for real token positions and $-\infty$ for padding positions
• $M_{\text{combined}}$: the result, which blocks a position if either mask blocks it

Since both masks use $-\infty$ for blocked positions and 0 for allowed positions, the sum gives $-\infty$ wherever either mask blocks attention. A position is only allowed (value 0) when both masks allow it: the position must be in the past (causal constraint) and must be a real token (padding constraint). This is the logical AND of the two constraints, implemented through simple arithmetic addition.

Why does addition work here? Because adding any finite number to $-\infty$ still gives $-\infty$, and adding zero to any value leaves it unchanged. The mask values act like boolean logic: 0 is "true" (allow) and $-\infty$ is "false" (block), and addition implements the OR operation on the "block" signal.

The green cells marked "Y" show where attention is allowed. For sequence 0, only the 2x2 lower-left block is active. For sequence 1, the lower-left triangular 3x3 region allows attention. Both patterns combine causal and padding constraints in a single mask. The visual pattern makes clear how the two masks compose: the causal mask "cuts" along the diagonal, and the padding mask "cuts" along the rightmost columns.

To make the mechanics concrete, let's trace through a complete masked attention computation step by step with small numerical values you can verify by hand.

We will use a toy sequence of three tokens: "I", "love", "AI". We want to apply causal masking so that during training, each position predicts the next token using only past context.

Step 1: Set up the scores. Suppose we have the following raw attention scores (these would normally come from the dot product of queries and keys, but we will use fixed values for clarity):

where $S_{ij}$ is the score from query position $i$ to key position $j$.

Step 2: Construct the causal mask. For a 3-token sequence, the causal mask is:

In practice we use $-10^9$ instead of $-\infty$.

Step 3: Add the mask to the scores. The masked scores are $S + M$:

The upper triangle has been driven to $-\infty$, making those positions invisible.

Step 4: Apply softmax row by row. Recall that softmax divides each exponential by the sum of exponentials in the row. For row 0 (position "I"):

Position "I" attends only to itself: weight 1.0. For row 1 (position "love"):

Position "love" splits its attention between "I" (40%) and itself (60%). For row 2 (position "AI"):

Position "AI" can attend to all three positions and distributes attention based on score magnitude, attending most strongly to itself (1.5 is the highest score).

Step 5: Interpret the result. The final attention weight matrix is:

Each row is a valid probability distribution summing to 1.0. The upper triangle is exactly zero. The model is now constrained to make predictions using only the causal context it should have. When predicting the word that follows "love," the model sees only "I" and "love"; when predicting the word after "AI," it sees all three. This is precisely the progressive left-to-right context structure that makes autoregressive language modeling work.

Attention masks can have different shapes depending on the use case. Understanding these shapes and how they broadcast determines whether the implementation works as intended.

The attention score matrix has shape (batch, num_heads, seq_len, seq_len) in a full transformer with multi-head attention. Masks can be provided in more compact forms and rely on NumPy or PyTorch's broadcasting rules to expand to the full shape automatically. The four common mask shapes are:

• Full shape (batch, num_heads, seq_len, seq_len): Fully specified mask for every batch item and head, giving maximum control but consuming the most memory
• Per-batch (batch, 1, seq_len, seq_len): Same mask across all heads for each batch item; the head dimension broadcasts to num_heads
• Per-batch compact (batch, 1, 1, seq_len): Key-only masking where the query dimension also broadcasts; this is the natural shape for padding masks, since padding depends only on which keys are padding, not on which query is asking
• Global (1, 1, seq_len, seq_len): Same mask for the entire batch; this is the natural shape for causal masks, since the causal constraint is the same for every sequence

Broadcasting allows us to store the mask in its most natural compact form and let the framework handle expansion. This is both memory-efficient and computationally clean: you define the mask at the level of abstraction that matches its semantics, and broadcasting handles the rest.

Broadcasting rules make mask combination elegant. The causal mask (1, 1, 8, 8) applies identically to all batch items and heads. The padding mask (2, 1, 1, 8) applies per batch item across all heads, masking keys based on which positions are padding. The final result (2, 4, 8, 8) has the correct combined mask for each batch item and attention head. Neither mask needs to be explicitly replicated to match the full score tensor shape; broadcasting handles that automatically and without allocating additional memory.

For very long sequences, storing full (seq_len, seq_len) masks becomes expensive. A 10,000-token sequence requires 100 million entries per mask. For a batch of 32 sequences with 4 attention heads, the full mask tensor would require 32 * 4 * 10,000 * 10,000 * 4 bytes = 51.2 GB, which is completely infeasible. Several strategies reduce this cost.

Lazy mask generation: Instead of precomputing the full mask, generate it during the attention computation.

For sequences of length 1000, boolean masks use 75% less memory. This adds up when processing large batches or very long documents. Modern deep learning frameworks like PyTorch support boolean masks natively and perform the float conversion internally only when needed, so you get the memory savings without manual conversion code.

With our understanding of padding and causal masks in place, we can now build a complete masked attention function. The key insight is that masking fits directly into the standard attention formula through a single addition operation before the softmax. No additional branches, no special-casing, no changes to the backpropagation graph: just one extra tensor addition that changes everything about which positions contribute.

Recall that standard scaled dot-product attention computes:

where:

• $Q$: the query matrix, shape $(n, d_k)$, where each row is a query vector for one position
• $K$: the key matrix, shape $(n, d_k)$, where each row is a key vector for one position
• $V$: the value matrix, shape $(n, d_v)$, where each row is a value vector for one position
• $d_k$: the dimension of the query and key vectors, used in the scaling factor $1/\sqrt{d_k}$

To add masking, we simply insert the mask matrix $M$ before the softmax:

where:

• $Q$: the query matrix of shape $(n, d_k)$, representing what each position is "looking for"
• $K$: the key matrix of shape $(n, d_k)$, representing what each position "offers"
• $V$: the value matrix of shape $(n, d_v)$, containing the information to aggregate
• $M$: the mask matrix of shape $(n, n)$, with 0 for allowed and $-\infty$ for blocked positions
• $d_k$: the dimension of queries and keys, used for scaling to prevent vanishing gradients
• $n$: the sequence length

The formula unfolds in three steps:

1. Compute raw scores: $QK^T$ produces an $(n, n)$ matrix where entry $(i, j)$ measures the similarity between query $i$ and key $j$.

2. Scale and mask: Divide by $\sqrt{d_k}$ to stabilize gradients, then add the mask $M$. Masked positions receive $-\infty$, which the softmax will convert to near-zero weights.

3. Normalize and aggregate: Softmax converts scores to weights summing to 1 (per row), then these weights select and combine values from $V$.

This formulation is elegant because the mask $M$ does not change the computational structure. We still compute all pairwise scores, but the masking happens through simple addition before softmax. The exponential function in softmax then "erases" the masked positions. The gradient with respect to masked positions is also effectively zero, so backpropagation naturally ignores them during parameter updates.

Let's translate the formula into code. The implementation follows the three-step structure exactly:

Notice how the mask application is just a single line: scores = scores + mask. This simplicity is intentional. The mask contains 0 for allowed positions (no effect on scores) and large negative values for blocked positions (which softmax converts to near-zero weights). No branching, no special cases, just addition. The mask is a first-class citizen of the attention formula, not a post-hoc patch.

Let's verify that our implementation correctly handles both padding and causal masks:

The attention weight matrix confirms that both masks work together correctly. Looking at the output for batch 1 (which has 4 real tokens and 2 padding tokens):

• Columns 4 and 5 are nearly zero: The padding mask successfully blocks attention to padding positions. No real token wastes attention on meaningless padding.

• Upper triangle is nearly zero: The causal mask blocks attention to future positions. Position 0 cannot see positions 1-5, position 1 cannot see positions 2-5, and so on.

• Lower-left region has non-zero weights: The intersection of "past positions" and "real tokens" receives all the attention. These are exactly the positions each query should attend to.

The combined mask creates a triangular pattern truncated by the padding boundary. This is precisely what a causal language model needs when processing variable-length batched sequences. The two masks, one enforcing temporal ordering and one enforcing content validity, combine into a single constraint.

Beyond standard padding and causal masks, researchers have explored various attention patterns for efficiency and modeling goals. Masking is a design tool as well as a correctness constraint: by choosing which positions can attend to which others, you shape what information the model can use and at what computational cost.

Think of the full attention matrix as a complete graph: every token is connected to every other token. Masking is the process of removing edges from this graph. The causal mask removes all "forward" edges. The padding mask removes all "noise" edges. Custom patterns remove different subsets of edges to achieve specific computational or modeling goals.

Restrict attention to a fixed window around each position. Instead of attending globally, each token only looks at its $w$ nearest neighbors on each side. This reduces complexity from $O(n^2)$ to $O(n \cdot w)$, where:

• $n$: the sequence length (total number of tokens)
• $w$: the window size (number of positions each token can attend to on each side)

The trade-off is expressiveness for efficiency. Local attention cannot capture dependencies between positions that are more than $w$ steps apart. For tasks where most relevant context is nearby, this is not a significant limitation. For tasks requiring long-range reasoning, it can be a serious bottleneck.

Local attention is the core mechanism in models like Longformer and BigBird designed to handle very long sequences. The window captures local context efficiently, and special "global" tokens (like [CLS]) can still attend to all positions for tasks that require long-range information. The combination of local and global tokens gives these models both efficiency and expressiveness.

Attend to every $k$-th position, spreading attention across the sequence with fixed intervals. Here, $k$ is the stride parameter that controls how far apart attended positions are. Where local attention captures fine-grained local context, strided attention captures coarse global patterns by sampling the sequence at regular intervals.

Strided attention is most useful when combined with local attention. By itself, strided attention misses the fine-grained context between strided positions. But paired with local attention, it provides the long-range "summary" connections that local windows cannot reach. This is the insight behind Sparse Transformer.

Real efficient attention mechanisms often combine multiple patterns. The Sparse Transformer uses local and strided attention together to cover both nearby and distant positions at a fraction of the cost of full attention.

The efficiency gains are dramatic. Full attention always uses 100% of possible pairs. Causal attention uses approximately 50% (the lower triangle). But local and sparse patterns become increasingly efficient as sequences grow longer: at 1024 tokens, local attention with window 32 uses only a few percent of pairs, and sparse patterns even less. This is why efficient attention variants are essential for processing long documents, where full attention would be computationally prohibitive.

Attention masking is essential infrastructure for modern transformers, but it also introduces implementation constraints and failure modes that practitioners need to understand.

The most significant computational limitation is that standard masking does not reduce the fundamental $O(n^2)$ cost of computing all pairwise attention scores. Even with half the positions masked, we still allocate and populate the full $n \times n$ score matrix before applying the mask. For causal attention, approximately half the matrix is immediately set to near-zero by softmax, but the memory for those entries and the computation to produce them has already been spent. Truly sparse attention requires specialized implementations that physically skip the masked computations, such as block-sparse matrix operations or the custom CUDA kernels used in FlashAttention. Writing and optimizing such kernels is significantly harder than dense matrix multiplication, which benefits from decades of hardware and software optimization. This gap between the theoretical efficiency of sparse attention and its practical implementation complexity is one of the active challenges in efficient transformer research.

Masking also fundamentally constrains what the model can learn, and the mask choice becomes an architectural decision with lasting consequences. Causal masking prevents bidirectional context: when processing the word "bank," the model cannot look ahead to "river" or "money" to resolve the ambiguity. This is by design for generative models, but it makes causal models weaker for discriminative tasks like sentiment classification or named entity recognition, where the full context is available and relevant. This is precisely why BERT was designed without causal masking, using bidirectional attention instead. The choice between causal and bidirectional attention is not a technical detail but an architectural philosophy that shapes the model's capabilities and the tasks it excels at.

A subtler limitation involves the interaction between masking and gradient flow during training. When entire rows of the attention matrix are masked (for instance, padding positions in the query dimension), the model has no gradient signal at those positions. This is correct behavior, but it means that models trained on heavily padded batches receive weaker gradient signals. In extreme cases, if most of a batch is padding, training can become unstable or slow. Practitioners typically bucket sequences by length before batching to minimize padding, but this adds preprocessing complexity and can introduce bias if done carelessly.

Despite these limitations, masking made several capabilities of modern NLP practical. Causal masking enabled efficient parallel training of autoregressive models, collapsing what would be $n$ sequential forward passes into a single parallel one. This made training on trillions of tokens computationally feasible and gave us the GPT family of models. Padding masks allowed practical batching of diverse real-world text, which is essential for any realistic training pipeline since natural language is inherently variable in length. Custom patterns like local and strided attention extended transformers to sequence lengths that would otherwise require prohibitive amounts of memory and compute, enabling applications in genomics, document understanding, and audio processing where sequences can be thousands or millions of tokens long.

The mask is also a safety mechanism. The causal mask is what prevents the model from developing a fundamentally incorrect inductive bias during training, one that exploits future tokens. A model trained without the causal mask on language modeling data would appear to perform well during training but would fail catastrophically at inference time, since future tokens are unavailable during generation. The mask enforces consistency between training and inference, and that consistency is foundational to reliable language modeling.

When implementing attention masking, several parameters control the mask behavior:

• mask_value: The large negative value used for masked positions (typically $-10^9$ or -float('inf')). Using $-10^9$ rather than true infinity avoids numerical issues with some operations while still producing near-zero attention weights after softmax. If every position in a row is masked, using $-\infty$ produces NaN via $0/0$ in softmax, while $-10^9$ produces a uniform distribution that is mathematically wrong but numerically stable.

• window_size (local attention): Controls how many positions on each side a token can attend to. Smaller windows reduce computation but may miss important long-range dependencies. Common values range from 128 to 512 tokens in production systems.

• stride (strided attention): Determines the spacing between attended positions in sparse patterns. A stride of $k$ means attending to every $k$-th position, reducing complexity while maintaining some global connectivity.

• pad_token_id: The token ID used for padding in tokenized sequences. This must match the padding token used during tokenization to correctly identify positions to mask.

Attention masking controls which positions can attend to which others, enabling autoregressive generation, efficient batch processing, and scalable attention over long sequences. The mechanism is remarkably simple: add large negative values to the scores of positions that should be blocked, and let softmax do the rest. The complexity lies not in the mechanism but in understanding when and why each type of mask is needed.

The key concepts from this chapter are:

• Additive masking: Adding large negative values before softmax drives attention weights to near-zero, effectively blocking those positions. No special logic is required in the attention function itself.
• Padding masks: Prevent attention to padding tokens when batching sequences of different lengths. Real tokens should not be influenced by meaningless padding values, and allowing such influence would corrupt representations and waste model capacity.
• Causal masks: Block attention to future positions, enforcing left-to-right information flow for autoregressive language models. This enables parallel training on full sequences while maintaining the correct sequential context constraint, collapsing $n$ sequential predictions into a single parallel forward pass.
• Mask combination: Multiple masks combine by addition. Any position blocked by any mask receives near-zero attention, implementing logical OR over the block conditions.
• Broadcasting: Masks can have shapes like (1, 1, seq_len, seq_len) for global patterns or (batch, 1, 1, seq_len) for per-sequence patterns, with broadcasting handling dimension expansion automatically and without extra memory allocation.
• Custom patterns: Local and strided attention, along with sparse patterns, reduce complexity for long sequences by limiting which positions can interact. These patterns trade some expressiveness for dramatic efficiency gains that make long-context modeling practical.
• Architectural choice: The choice of mask shapes what the model can learn. Causal masking suits generative models; bidirectional (no causal mask) attention suits discriminative models. This decision propagates through all downstream fine-tuning and deployment choices.

In the next chapter, we'll explore multi-head attention, which runs multiple attention operations in parallel with different learned projections. This allows the model to attend to information from different representation subspaces at different positions simultaneously, a capability that masking makes possible to apply selectively and efficiently.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about attention masking.