Decoder Architecture › The Causal Masking Requirement

To understand why causal masking is essential, consider what happens without it. In standard self-attention, every position can attend to every other position. When processing the word "cat" in "The cat sat on the mat," the model can look at "sat," "on," "the," and "mat" to help represent "cat." This bidirectional view is powerful for understanding, but it creates a fundamental problem for generation.

During generation, when we're about to predict "sat," the words "on," "the," and "mat" don't exist yet. If the model learned to rely on future context during training, it would fail at inference time. Even worse, during training on complete sequences, the model could "cheat" by peeking at the answer. Position 3 could look at position 4 to see what word should come next, trivially solving the prediction task without learning meaningful patterns.

The solution is to enforce a strict temporal ordering: position $t$ can only attend to positions $0, 1, \ldots, t$. This constraint, called causal masking or autoregressive masking, ensures the model learns to predict using only past context.

How do we mathematically enforce "only attend to the past"? We need to modify the attention mechanism so that certain query-key pairs produce zero attention weight. The key insight is that attention weights come from a softmax over scores. If we can make certain scores extremely negative before the softmax, they'll become negligible in the final weights.

Standard scaled dot-product attention computes:

$$\text{Attention}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) = \text{softmax}\left(\frac{\mathbf{Q}\mathbf{K}^T}{\sqrt{d_k}}\right)\mathbf{V}$$

The term $\mathbf{Q}\mathbf{K}^T$ produces an $n \times n$ matrix of scores, where entry $(i, j)$ measures how much query $i$ should attend to key $j$. To block future positions, we add a mask matrix $\mathbf{M}$ to these scores before the softmax:

$$\text{Attention}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) = \text{softmax}\left(\frac{\mathbf{Q}\mathbf{K}^T}{\sqrt{d_k}} + \mathbf{M}\right)\mathbf{V}$$

where:

• $\mathbf{Q} \in \mathbb{R}^{n \times d_k}$: query matrix containing $n$ query vectors, each of dimension $d_k$
• $\mathbf{K} \in \mathbb{R}^{n \times d_k}$: key matrix containing $n$ key vectors, each of dimension $d_k$
• $\mathbf{V} \in \mathbb{R}^{n \times d_v}$: value matrix containing $n$ value vectors, each of dimension $d_v$
• $\mathbf{Q}\mathbf{K}^T \in \mathbb{R}^{n \times n}$: the raw attention score matrix, where entry $(i, j)$ measures similarity between query $i$ and key $j$
• $\sqrt{d_k}$: scaling factor that prevents dot products from growing too large as dimension increases
• $\mathbf{M} \in \mathbb{R}^{n \times n}$: the causal mask matrix that blocks future positions

The mask $\mathbf{M}$ must satisfy a simple requirement: add 0 to scores we want to keep, and add a very large negative number to scores we want to block. Formally:

$$M_{ij} = \begin{cases} 0 & \text{if } j \leq i \\ -\infty & \text{if } j > i \end{cases}$$

where:

• $i$: the query position (row index), representing the token doing the attending
• $j$: the key position (column index), representing the token being attended to
• $M_{ij}$: the mask value added to the attention score at position $(i, j)$

The condition $j \leq i$ captures "key is at or before query," which means the token at position $j$ is in the past (or present) relative to position $i$. When this holds, we add 0, leaving the score unchanged. When $j > i$, the key is in the future, so we add $-\infty$ to block that attention path.

This creates a characteristic triangular pattern:

• Row 0 (first token): only column 0 is allowed (the token can only see itself)
• Row 1 (second token): columns 0 and 1 are allowed (can see itself and the first token)
• Row $n-1$ (last token): all columns 0 through $n-1$ are allowed (can see the entire sequence)

The mathematical trick is elegant. After adding the mask, we apply softmax to convert scores to attention weights. For a masked position:

$$\text{softmax}(s_{ij} + (-\infty)) = \frac{e^{s_{ij} - \infty}}{\sum_k e^{s_{ik} + M_{ik}}}$$

As the mask value approaches $-\infty$, the numerator $e^{-\infty}$ approaches 0. The denominator remains positive because unmasked positions contribute finite, positive exponentials. The result:

$$\frac{e^{-\infty}}{\text{positive sum}} = \frac{0}{\text{positive sum}} = 0$$

Those positions receive exactly zero attention weight. The token at position $i$ effectively cannot see any token at positions $i+1, i+2, \ldots, n-1$. In practice, we use a large finite value like $-10^9$ instead of true infinity, which achieves the same effect with standard floating-point arithmetic.

Let's implement causal masking and visualize the resulting attention pattern. The mask is simply an upper triangular matrix of negative infinities: