Quadratic Attention Bottleneck › Key Parameters

When analyzing attention complexity or estimating resource requirements, these parameters directly determine memory and compute costs:

• n (sequence length): The number of tokens in the input. This is the most critical parameter since both memory and compute scale with $n^2$. Typical values range from 512 (early BERT) to 128K+ (modern long-context models). Even modest increases in $n$ have dramatic effects on resource consumption.

• d (model dimension): The embedding dimension of the model. Common values include 768 (GPT-2 Small), 4,096 (LLaMA 7B), and 8,192 (larger models). Complexity scales linearly with $d$, making it far less impactful than sequence length.

• h (number of heads): The number of parallel attention heads. Each head stores an independent $n \times n$ attention matrix, so memory scales linearly with $h$. Typical values range from 12 to 128.

• L (number of layers): The depth of the transformer. Memory for attention matrices scales linearly with layer count. A 24-layer model uses twice the attention memory of a 12-layer model.

• dtype_bytes: Precision of floating-point representation. Using fp16 (2 bytes) instead of fp32 (4 bytes) halves memory requirements and often doubles throughput on modern GPUs.

• overhead_factor: When estimating maximum sequence length, this represents the fraction of GPU memory available for attention matrices after accounting for model weights and activations, plus gradients. Typical values are 0.5 for inference and 0.2 for training.