Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Attention Complexity › The Attention Bottleneck

Michael BrenndoerferUpdated May 29, 20255 min read

Language AI HandbookAll chapters

Part 13 of 67Self-Attention

Chapter 95Attention Complexity › The Attention Bottleneck

Reading section 6 of 11 · The Attention Bottleneck · 5 minSection 6 of 11 · 5 min

Section 6 of 11 · 5 min read

Attention Complexity › The Attention BottleneckOne section · Sections in this chapter6 / 11 ↓

Sections in this chapter11

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The combination of quadratic time and space complexity creates what is known as the attention bottleneck. As we push for longer context windows, attention becomes the dominant cost, often exceeding all other components combined. This section makes that claim precise by computing the actual FLOP distribution across transformer components at various sequence lengths.

Understanding this distribution is important because it reveals where optimization effort should be directed. At short sequences, optimizing attention provides minimal benefit; the feed-forward network is the bottleneck and any attention speedup is drowned out. At long sequences, the situation reverses completely: you could make the feed-forward network infinitely fast and still be limited by attention's quadratic cost. Knowing which regime you are in determines whether efficient attention variants are worth the implementation complexity.

The crossover happens gradually. Think of a transformer layer as a budget of operations. At short sequences, the $n^2 d$ attention cost is small relative to the $n d^2$ cost of two linear layers in the feed-forward network. As sequence length grows, the attention cost grows quadratically while the feed-forward cost grows only linearly. At some crossover sequence length, attention overtakes the feed-forward as the dominant component, and beyond that point, attention efficiency is what matters most.

In[22]:

Code

def layer_component_flops(n, d, ffn_mult=4):
    """
    Break down FLOPs by transformer layer component.

    Args:
        n: Sequence length
        d: Model dimension
        ffn_mult: FFN hidden dimension multiplier (typically 4)

    Returns:
        Dict of FLOPs per component
    """
    # Attention
    attention_flops = 2 * n * n * d  # Q@K^T and attention@V
    qkv_proj_flops = 3 * n * d * d  # Q, K, V projections
    output_proj_flops = n * d * d  # Output projection

    # Feed-forward network
    ffn_hidden = d * ffn_mult
    ffn_flops = 2 * n * d * ffn_hidden  # Two linear layers

    return {
        "attention_core": attention_flops,
        "qkv_projection": qkv_proj_flops,
        "output_projection": output_proj_flops,
        "feed_forward": ffn_flops,
    }


# Analyze at different sequence lengths
d = 768
analysis_results = []
for n in [256, 512, 1024, 2048, 4096]:
    flops = layer_component_flops(n, d)
    total = sum(flops.values())
    analysis_results.append(
        {
            "n": n,
            "attention_pct": (flops["attention_core"] / total) * 100,
            "projections_pct": (
                (flops["qkv_projection"] + flops["output_projection"]) / total
            )
            * 100,
            "ffn_pct": (flops["feed_forward"] / total) * 100,
        }
    )

Out[23]:

Console

Transformer Layer FLOP Distribution (d=768):

   Seq Len    Attention    Projections        FFN
--------------------------------------------------
       256         5.3%          31.6%      63.2%
       512        10.0%          30.0%      60.0%
      1024        18.2%          27.3%      54.5%
      2048        30.8%          23.1%      46.2%
      4096        47.1%          17.6%      35.3%

The percentages reveal how computational burden shifts with sequence length. At 256 tokens, attention accounts for only about 5% of layer computation, with the feed-forward network dominating at around 69%. By 1024 tokens, attention grows to roughly 17%, and at 4096 tokens it reaches approximately 45%, nearly matching the FFN. The projection layers (QKV and output) remain relatively stable since they scale linearly with $n$ . This distribution explains why optimizing attention becomes critical for long-context applications while being less important for short sequences.

In practice, this means that for tasks like sentiment classification or named entity recognition with inputs under 512 tokens, attention optimizations provide almost no benefit. But for tasks like document summarization, code generation over large codebases, or multi-turn conversation with long history, attention is the dominant cost and any improvement there directly translates to faster, cheaper systems. The research on efficient transformers is driven almost entirely by the long-context use cases where this bottleneck is acute.

Out[24]:

Visualization

Stacked area chart showing attention growing from 20% to 50% of computation as sequence length increases. — Distribution of FLOPs across transformer layer components as sequence length increases. Attention core (the n^2 operation) grows to dominate at longer sequences, while feed-forward layers shrink proportionally.

This visualization shows the attention bottleneck in action. The red region (attention core) expands as sequence length grows, squeezing out other components. This is why efficient attention mechanisms are such an active research area. Notice that the transition is not sharp: even at 4096 tokens, feed-forward operations still account for a substantial fraction of computation. But the trend is clear, and at 8192 tokens and beyond, attention has become the defining cost.