Sliding Window Attention › Sliding Window Attention

Standard attention computes all $n^2$ pairwise interactions between tokens, but most of that computation may be wasted. In natural language, context is often local: the meaning of a word depends heavily on its immediate neighbors. Consider the phrase "the quick brown fox jumps over the lazy dog." When determining what "jumps" means, the model needs "fox" (the subject) and "over" (what follows), but distant tokens like "the" or "dog" contribute less to understanding this particular word. Sliding window attention exploits this locality by restricting each token to attend only to tokens within a fixed-size window around it.

The quadratic cost of full attention creates a concrete systems problem. For a sequence of 4,096 tokens, the attention matrix has over 16 million entries. For 32,768 tokens, that number exceeds a billion. Modern GPU memory is finite. The memory required to store these matrices, along with the compute needed to form them and backpropagate through them, becomes the primary bottleneck for long-context processing. Sliding window attention sidesteps this wall by making a principled bet: for most tokens, only nearby context matters, and any information from far away can reach the token indirectly through multiple layers.

This bet turns out to be surprisingly well-justified. When researchers measured how attention weight is distributed in trained transformers, they found that most heads concentrate their mass within a few dozen positions of the query. The long tail of distant attention weights was often near zero, suggesting that full attention was doing significant unnecessary work. Sliding window attention makes this empirical observation into a hard architectural constraint, trading the flexibility of occasionally useful long-range connections for a guaranteed linear cost.

This chapter explores how sliding window attention works, why it reduces complexity from quadratic to linear, and how to implement it effectively. We will also examine the trade-offs: what do we lose when tokens cannot see beyond their local window, and how can we address these limitations?