Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Longformer › Visualizing the Attention Pattern

Michael BrenndoerferUpdated June 25, 20253 min read

Language AI HandbookAll chapters

Part 17 of 67Efficient Attention

Chapter 121Longformer › Visualizing the Attention Pattern

Reading section 4 of 16 · Visualizing the Attention Pattern · 3 minSection 4 of 16 · 3 min

Section 4 of 16 · 3 min read

Longformer › Visualizing the Attention PatternOne section · Sections in this chapter4 / 16 ↓

Sections in this chapter16

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The Longformer attention pattern creates a distinctive sparse structure when visualized as a matrix. The sparsity pattern is not random. It has a specific geometry: a diagonal band of width $w$ (the local window) with full rows and columns at the global token positions. When you see this pattern, you can immediately read off the design intent of the model. The density of the diagonal band tells you how much local context each token accesses. The number of full rows tells you how many global aggregation points the task requires. Building intuition for this visual representation will help you debug attention configurations and understand why certain configurations work better than others.

Let's build a visualization to understand this pattern in detail.

In[6]:

Code

import numpy as np


def create_longformer_attention_mask(seq_len, window_size, global_positions):
    """
    Create a Longformer-style attention mask.

    Args:
        seq_len: Total sequence length
        window_size: Size of sliding window (must be even)
        global_positions: List of positions that have global attention

    Returns:
        Attention mask of shape (seq_len, seq_len)
    """
    mask = np.zeros((seq_len, seq_len))
    half_window = window_size // 2

    # Add sliding window attention for all positions
    for i in range(seq_len):
        start = max(0, i - half_window)
        end = min(seq_len, i + half_window + 1)
        mask[i, start:end] = 1

    # Add global attention: global tokens attend to all, all attend to global
    for g in global_positions:
        mask[g, :] = 1  # Global token attends to all
        mask[:, g] = 1  # All tokens attend to global token

    return mask

Out[7]:

Visualization

Heatmap showing Longformer attention mask with diagonal sliding window band and cross-shaped global attention stripes. — Longformer attention pattern with window size 8 and global attention at positions 0 and 32. The diagonal band represents sliding window attention, while the vertical and horizontal stripes show global token connectivity. This sparse pattern reduces complexity from O(n²) to O(n) while maintaining long-range information flow.

The visualization reveals the structure of Longformer attention. The diagonal band represents sliding window attention: each position attends to its local neighborhood. The vertical and horizontal stripes at positions 0 and 32 show global attention: these tokens have bidirectional access to the entire sequence.

Notice how sparse this pattern is compared to full attention. Full attention would fill the entire square, requiring storage for $n^2 = 64^2 = 4{,}096$ attention weights (where $n = 64$ is our sequence length in this example). The Longformer pattern only needs storage for the non-zero entries, which scales linearly with sequence length.