Attention Masking › Custom Attention Patterns
Language AI HandbookAll chapters Attention Masking › Custom Attention PatternsOne section · Sections in this chapter9 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Beyond standard padding and causal masks, researchers have explored various attention patterns for efficiency and modeling goals. Masking is a design tool as well as a correctness constraint: by choosing which positions can attend to which others, you shape what information the model can use and at what computational cost.
Think of the full attention matrix as a complete graph: every token is connected to every other token. Masking is the process of removing edges from this graph. The causal mask removes all "forward" edges. The padding mask removes all "noise" edges. Custom patterns remove different subsets of edges to achieve specific computational or modeling goals.
Local Attention
Restrict attention to a fixed window around each position. Instead of attending globally, each token only looks at its nearest neighbors on each side. This reduces complexity from to , where:
- : the sequence length (total number of tokens)
- : the window size (number of positions each token can attend to on each side)
The trade-off is expressiveness for efficiency. Local attention cannot capture dependencies between positions that are more than steps apart. For tasks where most relevant context is nearby, this is not a significant limitation. For tasks requiring long-range reasoning, it can be a serious bottleneck.
def create_local_attention_mask(seq_len, window_size):
"""
Create a local attention mask with a fixed window.
Each position attends to window_size positions before and after.
Args:
seq_len: sequence length
window_size: number of positions to attend on each side
Returns:
mask: (seq_len, seq_len) attention mask
"""
mask = np.full((seq_len, seq_len), -1e9)
for i in range(seq_len):
start = max(0, i - window_size)
end = min(seq_len, i + window_size + 1)
mask[i, start:end] = 0.0
return mask
seq_len = 10
window_size = 2
local_mask = create_local_attention_mask(seq_len, window_size)
Local attention is the core mechanism in models like Longformer and BigBird designed to handle very long sequences. The window captures local context efficiently, and special "global" tokens (like [CLS]) can still attend to all positions for tasks that require long-range information. The combination of local and global tokens gives these models both efficiency and expressiveness.
Strided Attention
Attend to every -th position, spreading attention across the sequence with fixed intervals. Here, is the stride parameter that controls how far apart attended positions are. Where local attention captures fine-grained local context, strided attention captures coarse global patterns by sampling the sequence at regular intervals.
def create_strided_attention_mask(seq_len, stride):
"""
Create a strided attention mask.
Each position attends to every stride-th position.
Args:
seq_len: sequence length
stride: attend to every stride-th position
Returns:
mask: (seq_len, seq_len) attention mask
"""
mask = np.full((seq_len, seq_len), -1e9)
for i in range(seq_len):
# Attend to positions at stride intervals, aligned to position
for j in range(0, seq_len, stride):
mask[i, j] = 0.0
# Also attend to local neighborhood for current position
mask[i, i] = 0.0
return mask
stride = 3
strided_mask = create_strided_attention_mask(seq_len, stride)Strided attention is most useful when combined with local attention. By itself, strided attention misses the fine-grained context between strided positions. But paired with local attention, it provides the long-range "summary" connections that local windows cannot reach. This is the insight behind Sparse Transformer.
Combined Patterns
Real efficient attention mechanisms often combine multiple patterns. The Sparse Transformer uses local and strided attention together to cover both nearby and distant positions at a fraction of the cost of full attention.
def create_sparse_attention_mask(seq_len, local_window, stride):
"""
Create a sparse attention mask combining local and strided patterns.
Args:
seq_len: sequence length
local_window: size of local attention window
stride: stride for global attention
Returns:
mask: (seq_len, seq_len) attention mask
"""
# Start with all blocked
mask = np.full((seq_len, seq_len), -1e9)
# Add local attention
for i in range(seq_len):
start = max(0, i - local_window)
end = min(seq_len, i + local_window + 1)
mask[i, start:end] = 0.0
# Add strided attention
for i in range(seq_len):
for j in range(0, seq_len, stride):
mask[i, j] = 0.0
return mask
sparse_mask = create_sparse_attention_mask(seq_len, local_window=1, stride=4)
Each pattern represents a different trade-off. Full attention has maximum expressiveness but cost. Sparse patterns reduce complexity at the cost of some long-range interactions. The choice depends on sequence length, available compute, and task requirements.
# Compare sparsity levels across different mask patterns
def compute_sparsity(mask):
"""Compute the fraction of allowed (non-masked) positions."""
return (mask == 0).sum() / mask.size
seq_lengths = [64, 128, 256, 512, 1024]
patterns = {
"Full": [],
"Causal": [],
"Local (w=32)": [],
"Sparse (w=16, s=64)": [],
}
for n in seq_lengths:
# Full attention: all n^2 pairs
patterns["Full"].append(1.0)
# Causal: lower triangle = n(n+1)/2 pairs
causal = np.tril(np.ones((n, n)))
patterns["Causal"].append(causal.sum() / causal.size)
# Local with window 32
local = create_local_attention_mask(n, window_size=32)
patterns["Local (w=32)"].append((local == 0).sum() / local.size)
# Sparse: local window 16 + stride 64
sparse = create_sparse_attention_mask(n, local_window=16, stride=64)
patterns["Sparse (w=16, s=64)"].append((sparse == 0).sum() / sparse.size)
The efficiency gains are dramatic. Full attention always uses 100% of possible pairs. Causal attention uses approximately 50% (the lower triangle). But local and sparse patterns become increasingly efficient as sequences grow longer: at 1024 tokens, local attention with window 32 uses only a few percent of pairs, and sparse patterns even less. This is why efficient attention variants are essential for processing long documents, where full attention would be computationally prohibitive.