Multi-Query Attention › MQA vs GQA

Michael BrenndoerferUpdated August 4, 20253 min read
Language AI HandbookAll chapters
Part 29 of 67Modern Decoder Models
Chapter 210Multi-Query Attention › MQA vs GQA
Reading section 7 of 11 · MQA vs GQA · 3 minSection 7 of 11 · 3 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 210 of 513

Section 7 of 11 · 3 min read

Multi-Query Attention › MQA vs GQAOne section · Sections in this chapter7 / 11 ↓
Sections in this chapter11
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Grouped Query Attention interpolates between MHA and MQA by grouping heads and sharing K/V within each group.

In[21]:
Code
def attention_comparison(num_heads: int) -> dict:
    """Compare MHA, MQA, and various GQA configurations."""
    configs = {
        "MHA": {"kv_heads": num_heads, "sharing_factor": 1},
        "GQA-8": {"kv_heads": num_heads // 4, "sharing_factor": 4},
        "GQA-4": {"kv_heads": num_heads // 8, "sharing_factor": 8},
        "GQA-2": {"kv_heads": 2, "sharing_factor": num_heads // 2},
        "MQA": {"kv_heads": 1, "sharing_factor": num_heads},
    }
    return configs


num_heads = 32
configs = attention_comparison(num_heads)
Out[22]:
Visualization
Diagram showing the spectrum of attention variants from MHA to MQA with GQA options in between.
The attention mechanism spectrum from full MHA to extreme MQA. GQA offers intermediate points that balance memory efficiency with model expressiveness. Most modern LLMs use GQA with 2-8 KV heads.
Out[23]:
Console
Attention Variant Comparison (32 query heads):

Variant      KV Heads     Sharing Factor   Cache Reduction 
--------------------------------------------------------
MHA          32           1                1x              
GQA-8        8            4                4x              
GQA-4        4            8                8x              
GQA-2        2            16               16x             
MQA          1            32               32x

The spectrum shows a smooth trade-off between expressiveness and efficiency. GQA-8 with 8 KV heads provides a 4x cache reduction while maintaining most of the representational power of full MHA. GQA-4 doubles the savings to 8x. MQA represents the extreme endpoint with maximum savings but minimum KV diversity.

Why GQA Often Wins

In practice, GQA with 4-8 KV heads often provides the best trade-off:

  1. Near-MQA memory efficiency: 4-8x cache reduction covers most memory constraints
  2. Near-MHA quality: Multiple KV representations preserve expressiveness for complex tasks
  3. Flexible tuning: Choose the balance point appropriate for your deployment

LLaMA-2 70B uses GQA with 8 KV heads (sharing among 8 query heads each), achieving excellent quality while maintaining reasonable inference costs. Mistral and many other modern models follow similar patterns.