Sliding Window Attention › The Locality Assumption
Language AI HandbookAll chapters Sliding Window Attention › The Locality AssumptionOne section · Sections in this chapter2 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Before diving into the mechanics, let's understand why local attention makes sense. Natural language exhibits strong locality: adjacent words form phrases, phrases form clauses, and meaning often crystallizes within a narrow span.
Sliding window attention restricts each token to attending only to tokens within a fixed distance (the window size). Instead of attention weights, we compute approximately weights, achieving linear complexity in sequence length.
Consider how you read this sentence. When your eye lands on "read," you immediately connect it to "you" (subject) and "this sentence" (object), all within a few tokens. You do not need to re-read the beginning of the paragraph to understand this verb. Sliding window attention captures this intuition by limiting attention to a local neighborhood.
The key insight is that attention patterns in trained transformers often concentrate locally. Research analyzing attention weights in models like BERT and GPT-2 found that many heads allocate most attention mass to nearby tokens. By building this locality bias into the architecture, we can dramatically reduce computation while preserving much of the model's representational power.
The locality assumption predates the transformer era. Local receptive fields drove convolutional neural networks in image processing throughout the 1990s and 2000s, where each filter looked at a small spatial patch rather than the whole image. When transformers arrived in 2017, researchers initially used full attention because sequence lengths were short enough to afford it. As models started processing longer inputs, the community began revisiting locality. The 2020 Longformer paper by Beltagy et al. was among the first to systematically combine sliding window attention with sparse global tokens for document-level NLP. This shows that local attention could match or exceed full attention on many benchmarks. Subsequent work in Mistral (2023) showed that even large language models could use windowed attention without meaningful quality loss, cementing it as a practical standard for efficient long-context inference.