Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Sliding Window Attention › Dilated Sliding Windows

Michael BrenndoerferUpdated June 26, 20256 min read

Language AI HandbookAll chapters

Part 17 of 67Efficient Attention

Chapter 119Sliding Window Attention › Dilated Sliding Windows

Reading section 5 of 12 · Dilated Sliding Windows · 6 minSection 5 of 12 · 6 min

Section 5 of 12 · 6 min read

Sliding Window Attention › Dilated Sliding WindowsOne section · Sections in this chapter5 / 12 ↓

Sections in this chapter12

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Standard sliding windows have a limitation: the receptive field grows slowly with window size. To see positions 100 tokens away, you need either a window of 200 tokens (expensive) or many layers for information to propagate (slow). Dilated sliding windows offer a third option: spread the attention across a wider range by skipping intermediate positions.

Dilated Attention

Dilated sliding window attention attends to positions at regular intervals (the dilation rate) rather than consecutive positions. This expands the effective receptive field while keeping the number of attended positions constant.

The intuition comes from signal processing and computer vision. A dilated convolution samples every $r$ th position instead of consecutive positions, covering more ground with the same number of parameters. Dilated attention applies the same principle: instead of attending to your 5 nearest neighbors, attend to 5 positions spaced 2 apart.

How Dilation WorksLink Copied

Let's formalize this idea. In standard attention, the attended positions form a consecutive sequence centered on the query. With dilation, we introduce gaps between attended positions.

With dilation rate $r$ and $k$ attended positions, token $i$ attends to the following set of positions:

\{i - r \cdot k/2, \, i - r \cdot (k/2 - 1), \, \ldots, \, i, \, \ldots, \, i + r \cdot (k/2 - 1), \, i + r \cdot k/2\}

where:

$i$ : the query position (the token computing its attention)
$r$ : the dilation rate (the gap between consecutive attended positions)
$k$ : the number of positions to attend to (same as in non-dilated attention)
$k/2$ : the number of positions on each side of the center

The dilation rate $r$ acts as a multiplier on the offset from the center position. With $r = 1$ (no dilation), we get the standard consecutive window. With $r = 2$ , we skip every other position, doubling the effective range while attending to the same number of positions.

For example, with $k = 5$ and $r = 2$ , token at position 10 attends to positions $\{6, 8, 10, 12, 14\}$ rather than the consecutive window $\{8, 9, 10, 11, 12\}$ . Both patterns attend to exactly 5 positions, but the dilated version covers a range of 8 positions instead of 4.

Out[12]:

Visualization

Attention matrix showing narrow diagonal band of consecutive positions. — Standard window (dilation=1): Attends to 5 consecutive positions.

Attention matrix showing wider diagonal pattern with gaps between attended positions. — Dilated window (dilation=2): Attends to 5 positions with gaps, covering twice the range.

Multi-Scale Attention with Varying DilationLink Copied

A single dilation rate creates gaps in coverage: with $r=2$ , we miss all odd-positioned tokens. The solution is to use different dilation rates across attention heads. Some heads use $r=1$ for fine-grained local attention, others use $r=2$ for medium-range context, and still others use $r=4$ or $r=8$ for long-range dependencies.

This multi-scale approach, used in models like Longformer, creates a powerful combination. The union of all heads covers a broad range of positions, each head specializes in a particular scale, and the total computation remains sparse. Think of it like having multiple photographers at an event: one captures close-ups, another medium shots, and a third wide angles. Together, they document everything without any single photographer doing all the work.

In[13]:

Code

def multi_scale_receptive_field(dilations, k):
    """
    Calculate coverage of multi-scale dilated attention.

    Args:
        dilations: List of dilation rates for different heads
        k: Number of positions each head attends to

    Returns:
        Total range covered by union of all heads
    """
    all_positions = set()
    center = 1000  # Arbitrary center position
    half_k = k // 2

    for d in dilations:
        for offset in range(-half_k, half_k + 1):
            all_positions.add(center + offset * d)

    min_pos = min(all_positions)
    max_pos = max(all_positions)
    return max_pos - min_pos, len(all_positions)


# Example: 4 heads with different dilations
dilations = [1, 2, 4, 8]
k = 5
range_covered, unique_positions = multi_scale_receptive_field(dilations, k)

Out[14]:

Console

Multi-scale dilated attention (k=5 positions per head):

Dilation rates: [1, 2, 4, 8]
Total range covered: 32 positions
Unique positions attended: 11
Coverage density: 33.3%

The multi-scale approach extends the effective receptive field while maintaining sparse attention. With dilation rates of $\{1, 2, 4, 8\}$ and 5 positions per head, the model covers a range of 32 positions with 11 unique attention connections, achieving both broad coverage and computational efficiency.

Out[15]:

Visualization

Five-row square-marker diagram showing four dilation heads and their combined attended positions, with overlapping coverage at the center. — Multi-scale dilated attention coverage from a single query position (center). Each head attends to 5 positions at different dilation rates. The bottom row shows combined coverage where darker colors indicate positions attended by multiple heads.