Sliding Window Attention › Window Size Selection
Language AI HandbookAll chapters Sliding Window Attention › Window Size SelectionOne section · Sections in this chapter4 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Choosing the right window size involves balancing efficiency against the ability to capture dependencies. Smaller windows are faster but may miss important long-range relationships. Larger windows capture more context but reduce the computational advantage.
Factors Influencing Window Size
Several considerations guide window size selection:
-
Task requirements: Classification tasks often need only local features, while question answering may require longer-range reasoning. Summarization tasks might need to connect information across paragraphs.
-
Model depth: With multiple layers, information can propagate across the sequence. In a model with layers and window size , information can theoretically travel positions in each direction through the network. This is because each layer allows a token to gather information from positions on each side, and stacking layers compounds this reach. A 12-layer model with window size 128 can propagate information across positions in each direction.
-
Computational budget: Larger windows cost more. If latency is critical, smaller windows enable faster inference.
-
Sequence length: The ratio of window size to sequence length matters. A 256-token window covers an 8K-token document poorly but might suffice for 1K-token inputs.
Empirical Guidance
Research on efficient transformers has explored various window sizes:
- Longformer: Uses window sizes of 256-512 tokens, combined with global attention on specific positions
- BigBird: Employs block sizes of 64 tokens with random and global attention patterns
- Mistral 7B: Uses a 4,096-token sliding window, covering substantial context for most applications
A useful heuristic is to start with a window size that captures typical sentence or paragraph lengths (128-512 tokens) and increase it if downstream task performance suffers.
def effective_receptive_field(num_layers, window_size):
"""
Calculate the maximum distance information can travel.
With L layers and window w, token i can receive information
from token j if |i - j| <= L * (w // 2) for symmetric windows.
"""
return num_layers * (window_size // 2)
# Example: how far can information travel?
window_sizes = [64, 128, 256, 512]
layer_counts = [6, 12, 24]Effective Receptive Field (positions):
Window 6 layers 12 layers 24 layers
------------------------------------------
64 192 384 768
128 384 768 1,536
256 768 1,536 3,072
512 1,536 3,072 6,144The receptive field table shows how layer depth compensates for narrow windows. A 64-token window in a 24-layer model allows information to propagate 768 positions, while the same window in a 6-layer model reaches only 192 positions. This explains why deeper models can often use smaller windows without sacrificing performance.
