Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Quadratic Attention Bottleneck › Practical Sequence Length Limits

Michael BrenndoerferUpdated June 21, 20255 min read

Language AI HandbookAll chapters

Part 17 of 67Efficient Attention

Chapter 117Quadratic Attention Bottleneck › Practical Sequence Length Limits

Reading section 6 of 13 · Practical Sequence Length Limits · 5 minSection 6 of 13 · 5 min

Section 6 of 13 · 5 min read

Quadratic Attention Bottleneck › Practical Sequence Length LimitsOne section · Sections in this chapter6 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Given the quadratic scaling, what are the practical limits for standard attention? The answer depends on hardware constraints and whether you are training or doing inference, and the distinction between these two regimes is important.

During inference, you need to store only the current attention matrix (and possibly key-value caches for fast autoregressive generation). During training, you must also store the attention matrices for every layer simultaneously, because the backward pass needs them to compute gradients. This difference means that training is far more memory-constrained than inference, and the practical sequence length limit during training can be several times lower than during inference.

The key constraint is whether the entire computation graph fits after accounting for every memory consumer, including the attention matrices. Model weights typically occupy a fixed amount of memory regardless of sequence length. Activations from feed-forward layers and residual connections scale linearly with sequence length. Gradients and optimizer states (in training) scale with parameter count. Attention matrices then add the $O(n^2)$ term on top of all this.

In[13]:

Code

def estimate_max_sequence_length(
    gpu_memory_gb, n_heads, n_layers, overhead_factor=0.5, dtype_bytes=2
):
    """
    Estimate maximum sequence length that fits in GPU memory.

    Args:
        gpu_memory_gb: Available GPU memory
        n_heads: Number of attention heads
        n_layers: Number of layers
        overhead_factor: Fraction of memory available for attention
                        (rest goes to weights, activations, gradients)
        dtype_bytes: Bytes per element

    Returns:
        Maximum sequence length
    """
    available_bytes = gpu_memory_gb * (1024**3) * overhead_factor

    # Memory dominated by attention matrices: n_heads * n^2 * dtype_bytes * n_layers
    # Solving: n^2 = available_bytes / (n_heads * dtype_bytes * n_layers)
    n_squared = available_bytes / (n_heads * dtype_bytes * n_layers)
    max_n = int(np.sqrt(n_squared))

    return max_n


gpu_configs = [
    {"name": "RTX 4090", "memory": 24},
    {"name": "A100 40GB", "memory": 40},
    {"name": "A100 80GB", "memory": 80},
    {"name": "H100 80GB", "memory": 80},
]

model = {"heads": 12, "layers": 12}  # GPT-2 Small style

Out[14]:

Console

Estimated Maximum Sequence Length (GPT-2 Small, batch_size=1):

            GPU     Memory    Inference     Training
----------------------------------------------------
       RTX 4090         24 GB        6,688        4,230
      A100 40GB         40 GB        8,635        5,461
      A100 80GB         80 GB       12,211        7,723
      H100 80GB         80 GB       12,211        7,723

These estimates reveal a stark reality. Even on an 80 GB GPU, training a standard transformer is limited to roughly 16K-23K tokens before memory runs out. Inference allows longer sequences since gradients are not stored, but even then, sequences beyond 50K tokens become challenging. Production models that claim 100K+ context windows use specialized techniques covered in later chapters: FlashAttention for memory-efficient attention computation, grouped-query attention for reducing key-value cache size, and sliding-window attention for focusing on local context.

The overhead factor in our estimates (0.5 for inference, 0.2 for training) is a rough approximation. In practice, the exact fraction available for attention varies considerably based on model size, batch size, gradient checkpointing configuration, and optimizer choice. But the qualitative picture remains: training is always more memory-constrained than inference, and the gap grows as you add more heads and layers.

Out[15]:

Visualization

Log-log plot showing one normalized quadratic curve for both memory and compute, with GPU memory limits marked for training and inference scenarios. — Memory and compute requirements as sequence length increases, normalized to their values at n=512. Both scale quadratically (the single purple curve represents both), but memory hits a hard wall while compute can in principle be spread across time by processing smaller batches. The vertical dashed lines mark where an 80 GB GPU runs out of memory for attention matrices under training and inference regimes. The shaded region beyond the training limit requires either memory-efficient attention or gradient checkpointing.

The shaded region represents sequence lengths that exceed GPU memory for training. While compute could theoretically handle longer sequences given enough time (by reducing batch size or using gradient accumulation), memory is a hard constraint that cannot be overcome without specialized techniques. This asymmetry explains why memory-efficient attention methods like FlashAttention matter in practice: they shift the memory limit rightward without changing the compute requirements, opening up sequence lengths that were previously inaccessible.