Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Longformer › Comparing Attention Sparsity

Michael BrenndoerferUpdated June 25, 20252 min read

Language AI HandbookAll chapters

Part 17 of 67Efficient Attention

Chapter 121Longformer › Comparing Attention Sparsity

Reading section 5 of 16 · Comparing Attention Sparsity · 2 minSection 5 of 16 · 2 min

Section 5 of 16 · 2 min read

Longformer › Comparing Attention SparsityOne section · Sections in this chapter5 / 16 ↓

Sections in this chapter16

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let's quantify the sparsity achieved by Longformer compared to full attention.

In[8]:

Code

def compute_attention_density(seq_len, window_size, num_global):
    """Compute the fraction of non-zero attention weights."""
    # Sliding window: each of n tokens attends to w tokens
    sliding_window_entries = seq_len * window_size

    # Global attention: each global token adds (n-1) new connections
    # (excluding the window entries we already counted)
    global_entries = num_global * (seq_len - window_size) * 2

    # Total possible entries
    total_possible = seq_len**2

    # Approximate density (some overlap, but gives the right idea)
    density = min(
        1.0, (sliding_window_entries + global_entries) / total_possible
    )
    return density


# Compare densities at different sequence lengths
seq_lengths = [512, 1024, 2048, 4096, 8192, 16384]
window_size = 512
num_global = 2

densities = [
    compute_attention_density(n, window_size, num_global) for n in seq_lengths
]
full_attention_memory = [n**2 for n in seq_lengths]
longformer_memory = [n * (window_size + 2 * num_global) for n in seq_lengths]

Out[9]:

Console

Sequence Length | Attention Density | Memory Ratio (Longformer/Full)
-----------------------------------------------------------------
           512 |           100.0% |      100.78%
          1024 |            50.2% |       50.39%
          2048 |            25.1% |       25.20%
          4096 |            12.6% |       12.60%
          8192 |             6.3% |        6.30%
         16384 |             3.1% |        3.15%

The numbers show the difference. At 512 tokens, Longformer uses about the same memory as full attention. But as sequence length grows, the savings become dramatic. At 4,096 tokens, Longformer uses only about 12% of the memory. At 16,384 tokens, it drops to around 3%. This is the power of linear versus quadratic scaling.