Sliding Window Attention › Summary

Sliding window attention addresses the quadratic complexity of standard attention by restricting each token to attend only within a local window. The core ideas and takeaways from this chapter:

• Linear complexity: By attending to a fixed window of $w$ positions, complexity drops from $O(n^2 d)$ to $O(n \cdot w \cdot d)$, enabling efficient processing of long sequences.

• Window patterns: Symmetric windows work for bidirectional models, while causal windows suit autoregressive generation. Asymmetric and dilated windows offer additional flexibility.

• Effective receptive field: Through multiple layers, information propagates beyond the immediate window. A model with $L$ layers and window $w$ has an effective receptive field of approximately $L \cdot (w/2)$ positions in each direction, allowing distant tokens to influence each other indirectly.

• Implementation efficiency: Memory-efficient implementations avoid materializing the full attention matrix, instead computing attention block by block.

• Rolling KV cache: For autoregressive generation, a circular buffer stores only the most recent $w$ key-value pairs. This provides constant memory usage regardless of sequence length.

• Trade-offs: The locality assumption sacrifices direct long-range attention. Tasks requiring connections across distant positions may need additional mechanisms like global attention tokens.

• Practical success: Models like Mistral demonstrate that sliding window attention with large windows (4,096+ tokens) provides excellent performance on most tasks while enabling efficient inference.

The next chapter explores global attention tokens, which complement sliding window attention by providing a mechanism for information to flow across the entire sequence, combining the efficiency of local attention with the expressiveness of global connectivity.