Sliding Window Attention › Worked Example

To make the mathematics concrete, let's trace through a small sliding window attention computation by hand. Consider a sequence of five tokens, numbered 0 through 4, with a symmetric window of size 3. This means each token attends to itself and one neighbor on each side, attending to at most three positions in total.

Think of the tokens as the words in the fragment "cats chase small mice quickly." Token 2 ("small") sits in the middle. With a window of 3, it attends to tokens 1 ("chase"), 2 ("small"), and 3 ("mice"). Tokens 0 ("cats") and 4 ("quickly") fall outside the window and receive no attention weight.

At the boundaries, the window clips naturally. Token 0 can only attend to tokens 0 and 1 because negative positions do not exist. Token 4 can only attend to tokens 3 and 4 because position 5 is beyond the sequence. This boundary clipping is not a special case; it follows directly from the mask formula by clamping indices to valid range.

Let's work through the mask matrix $M$ explicitly. For $n = 5$ and $w = 3$, the allowed positions form a $5 \times 5$ binary grid where entry $(i, j)$ is 1 if $|i - j| \leq 1$:

$$M_{\text{binary}} = \begin{bmatrix} 1 & 1 & 0 & 0 & 0 \\ 1 & 1 & 1 & 0 & 0 \\ 0 & 1 & 1 & 1 & 0 \\ 0 & 0 & 1 & 1 & 1 \\ 0 & 0 & 0 & 1 & 1 \end{bmatrix}$$

where:

• $1$: attention is allowed between this query-key pair
• $0$: attention is blocked (the softmax mask receives $-\infty$ here)

Notice that the matrix is symmetric and banded. The main diagonal and the two adjacent diagonals are active. All entries more than one step away from the diagonal are zero. This banded structure is the defining visual signature of sliding window attention.

Now suppose we compute raw attention scores $QK^T / \sqrt{d_k}$ and get some matrix $S$. We add $M_{\text{additive}}$ where allowed positions receive 0 and blocked positions receive $-10^9$. After row-wise softmax, each row's weights sum to 1 but are supported only on the active positions in that row. Token 2's output is therefore a weighted blend of the representations of tokens 1, 2, and 3, with zero contribution from tokens 0 and 4.

The key insight is that no information is ever destroyed; it simply cannot flow directly across more than one window-width in a single layer. Across multiple layers, this local mixing accumulates. After two layers with a window of 3, token 2's representation has absorbed information from tokens 0 through 4 indirectly, because token 1 already incorporated token 0 in the first layer. The receptive field expands with depth, even though each individual attention operation remains strictly local.