Attention Complexity › Key Parameters for Complexity Analysis

When analyzing or estimating attention complexity, several parameters directly impact computational and memory requirements. Understanding how each one affects cost allows you to make informed choices when designing or selecting models.

• n (sequence length): The number of tokens in the input sequence. This is the most critical parameter since complexity scales quadratically with it. Typical values range from 512 to 128K+ in modern models. Because of the quadratic relationship, changes to this parameter have a much larger effect on total cost than changes to any other. Changing $n$ from 2K to 8K quadruples attention cost; no other parameter has this multiplier effect.

• d (model dimension): The embedding dimension of the model. Common values include 768 (GPT-2 Small), 1024 (GPT-2 Medium), 4096 (LLaMA 7B), and larger for bigger models. Complexity scales linearly with this parameter. Making models wider is relatively cheap compared to making them longer-context.

• h (number of heads): The number of attention heads. Each head operates on $d/h$ dimensions. More heads mean more parallel attention patterns but also more attention matrices to store. Critically, the total FLOPs for multi-head attention are the same as for a single head with the full dimension, because the reduction in per-head dimension exactly offsets the increase in number of heads. However, memory scales with $h$ because all $h$ attention matrices must be stored simultaneously.

• L (number of layers): Total transformer layers. Memory and compute scale linearly with layer count, so a 24-layer model requires twice the resources of a 12-layer model. Adding layers is one of the "cheap" ways to increase model capacity: it costs linearly and does not interact with the quadratic sequence-length term.

• dtype_bytes: Precision of floating-point representation. fp32 uses 4 bytes, fp16/bf16 use 2 bytes. Half precision halves memory requirements and often doubles throughput on modern GPUs. Moving from fp32 to bf16 is one of the simplest ways to double your effective context capacity within a fixed memory budget.

• batch_size: Number of sequences processed simultaneously. Memory scales linearly with batch size, often requiring batch size reduction for long sequences. At very long context lengths, batch sizes of 1 or 2 are common in production, meaning the full GPU memory is devoted to a single request's attention matrices and KV cache.