Sparse Attention Patterns › Combining Sparse Patterns
Language AI HandbookAll chapters Sparse Attention Patterns › Combining Sparse PatternsOne section · Sections in this chapter6 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Real-world efficient attention mechanisms combine multiple patterns to balance local context, global reach, and computational efficiency. The Sparse Transformer paper introduced the idea of factorizing attention across multiple heads, with different heads using different patterns.
The motivation for combining patterns comes from recognizing that no single sparse pattern dominates all task types. Local attention excels at capturing short-range syntactic structure but fails at long-range coreference. Strided attention provides long-range connectivity but may miss fine-grained local interactions between adjacent tokens. Block-sparse attention is hardware-efficient but requires careful alignment of content with block boundaries. By combining patterns, you can cover each other's blind spots.
The combination can happen in two fundamentally different ways. The first approach takes the union of two masks: a position pair is allowed if it is allowed by either pattern. This gives you the benefits of both patterns in every head. The disadvantage is that the combined mask is denser than either individual mask, increasing computation. The second approach assigns different patterns to different heads: local heads capture nearby context while strided heads capture long-range context. This keeps each head sparse and adds no overhead per head, but it requires enough heads to cover the pattern space. Models with 8 or 16 heads can dedicate 4 to local attention and 4 to strided attention without sacrificing capacity. Both strategies work in practice, and the choice often comes down to the number of available attention heads and the relative importance of local versus global relationships in the target task.
Local + Strided Combination
The most common combination pairs local attention for nearby tokens with strided attention for long-range connections. Together, they ensure every pair of positions can communicate within a small number of layers while maintaining overall sparsity.
def create_combined_sparse_mask(seq_len, local_window, stride):
"""
Combine local and strided attention patterns.
Args:
seq_len: Sequence length
local_window: Window size for local attention
stride: Stride for global attention
Returns:
mask: Combined boolean mask (OR of local and strided)
"""
local_mask = create_local_attention_mask(seq_len, local_window)
strided_mask = create_strided_attention_mask(seq_len, stride)
# Combine with OR: attend if either pattern allows
combined = local_mask | strided_mask
return combined, local_mask, strided_mask
seq_len = 48
combined, local, strided = create_combined_sparse_mask(
seq_len, local_window=3, stride=6
)
Pattern analysis (seq_len=48): Pattern Pairs Sparsity ---------------------------------------- Full 2,304 0.0% Local (w=3) 324 85.9% Strided (s=6) 744 67.7% Combined 930 59.6% Note: Combined has fewer pairs than sum of components due to overlap.
Multi-Head Factorization
An elegant approach from Sparse Transformer assigns different patterns to different attention heads. Half the heads might use local attention while the other half use strided attention. This factorization allows the model to learn which pattern is most useful for different types of dependencies.
def create_factorized_attention(seq_len, num_heads, local_window, stride):
"""
Create factorized attention patterns for multi-head attention.
Half the heads use local attention, half use strided attention.
Args:
seq_len: Sequence length
num_heads: Number of attention heads
local_window: Window for local heads
stride: Stride for strided heads
Returns:
masks: List of (seq_len, seq_len) masks, one per head
"""
local_mask = create_local_attention_mask(seq_len, local_window)
strided_mask = create_strided_attention_mask(seq_len, stride)
masks = []
for h in range(num_heads):
if h < num_heads // 2:
masks.append(local_mask)
else:
masks.append(strided_mask)
return masks
# Create masks for 8 heads
num_heads = 8
head_masks = create_factorized_attention(
seq_len, num_heads, local_window=3, stride=6
)Factorized attention (8 heads): Heads 0-3: Local attention (window=3) Heads 4-7: Strided attention (stride=6) Average pairs per head: 534 Full attention would be: 2,304 Overall sparsity: 76.8%
Factorized attention provides flexibility: local heads handle nearby dependencies while strided heads capture long-range patterns. The model learns to route information through the appropriate heads during training.
The factorization approach has an elegant information-theoretic justification. In standard multi-head attention, each head learns to focus on different types of relationships: some heads attend to syntactic structure, others to semantic similarity, others to positional patterns. By explicitly assigning structural patterns to heads at the architectural level, you are encoding inductive biases that align with these empirically observed behaviors. A head that can only attend locally is forced to specialize in local patterns; a head that can only attend at stride positions must specialize in global patterns. This constraint acts as a form of regularization, reducing the space of possible attention behaviors each head must search during training.