Attention Masking › Attention Masking

Self-attention computes pairwise interactions between all positions in a sequence, letting every token gather information from every other token. This generality is one of attention's greatest strengths, but it also raises an immediate practical problem: sometimes we need to block certain interactions entirely. A model generating text one token at a time must not be allowed to peek at the tokens that come after the current position. Sequences padded to a uniform length so they can be batched together should not let real tokens attend to meaningless padding. Controlling which positions are "visible" to which others is not a minor detail but a foundational design decision that shapes what a transformer can and cannot do.

Think of attention masking as a privacy screen on a monitor. Without the screen, anyone walking past can read what is displayed. With the screen, only people sitting in the correct position see the content, while everyone else sees darkness. In the same way, an attention mask designates which query-key pairs are "in view" (allowed to interact) and which are "blocked" (prevented from interacting). The screen does not change the underlying pixels; it simply controls visibility. The mask does not change the attention mechanism; it simply controls which scores contribute to the final weighted sum.

Masking modifies the attention computation by adding large negative values to specific positions before the softmax. Recall that attention weights are computed as:

$$\alpha_{ij} = \frac{\exp(s_{ij})}{\sum_{k=1}^{n} \exp(s_{ik})}$$

where:

• $\alpha_{ij}$: the attention weight from query position $i$ to key position $j$, representing how much position $i$ "attends to" position $j$
• $s_{ij}$: the raw attention score between positions $i$ and $j$, computed as the dot product of their query and key vectors (scaled by $1/\sqrt{d_k}$)
• $n$: the sequence length

When we add a large negative value (like $-10^9$) to a score $s_{ij}$, the exponential $\exp(s_{ij} - 10^9)$ becomes vanishingly small, driving $\alpha_{ij}$ to near-zero. These masked positions effectively disappear from the weighted sum. This simple mechanism enables causal language modeling, efficient batch processing, and custom attention patterns.

The elegance of this approach is that it requires no changes to the attention mechanism itself. The formula remains identical. The mask is simply an extra additive term inserted at one specific point in the computation, yet that term determines which positions survive softmax. Masked positions contribute nothing to the output, so from the model's perspective, they simply do not exist during that forward pass.

Understanding attention masking deeply matters for several reasons. First, it is the difference between a model that cheats during training (by reading future tokens) and one that learns to predict. Second, it is what makes batched training of variable-length sequences possible without corrupting representations. Third, it is the foundation of efficient attention variants that scale transformers to sequences far longer than standard full attention can handle. Every serious transformer practitioner needs to understand that masks exist, how they work, when to apply them, and what happens when they are misconfigured.