Multi-Query Attention › Summary
Language AI HandbookAll chapters Multi-Query Attention › SummaryOne section · Sections in this chapter9 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Multi-Query Attention addresses a key bottleneck in large language model deployment: KV cache memory consumption. By sharing a single key and value projection across all attention heads, MQA achieves substantial memory reductions:
- Memory reduction: KV cache shrinks by a factor of (the number of heads), often 32-64x for modern models
- Throughput gains: Smaller caches enable larger batch sizes and longer contexts on the same hardware
- Modest quality loss: Typically 1-3% degradation on standard benchmarks
- Parameter savings: K and V projection parameters reduce by a factor of
The trade-off between efficiency and expressiveness has led to the popular Grouped Query Attention (GQA), which shares K/V among groups of heads rather than all heads. Most modern LLMs, including LLaMA-2, Mistral, and others, use GQA with 4-8 KV heads as a practical middle ground.
For inference-heavy deployments where throughput matters more than the last percentage point of accuracy, MQA and GQA are essential techniques. They transform the economics of serving large language models, making long-context generation feasible on hardware that would otherwise be memory-bound.