Grouped Query Attention › GQA vs MQA: Quality Comparison
Language AI HandbookAll chapters Grouped Query Attention › GQA vs MQA: Quality ComparisonOne section · Sections in this chapter6 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Does GQA's additional key-value capacity translate to meaningful quality improvements over MQA? Research from Meta's LLaMA 2 paper and subsequent work provides empirical guidance.
The original GQA paper (Ainslie et al., 2023) conducted extensive experiments comparing MHA, GQA with various group sizes, and MQA. Key findings include:
- GQA with 8 KV heads matches MHA quality within 0.5% on most benchmarks while giving 8x memory reduction
- MQA shows 1-3% quality degradation compared to MHA on complex reasoning tasks
- The quality gap between GQA-8 and MQA is most pronounced on multi-hop reasoning and long-context retrieval
# Simulated benchmark results based on published research
benchmark_results = {
"model_configs": [
{"name": "MHA (baseline)", "kv_heads": 32, "relative_quality": 100.0},
{"name": "GQA-8", "kv_heads": 8, "relative_quality": 99.5},
{"name": "GQA-4", "kv_heads": 4, "relative_quality": 99.0},
{"name": "GQA-2", "kv_heads": 2, "relative_quality": 98.0},
{"name": "MQA", "kv_heads": 1, "relative_quality": 96.5},
],
"task_breakdown": {
"General LM": [100.0, 99.8, 99.5, 99.0, 98.0],
"Reasoning": [100.0, 99.2, 98.5, 97.0, 95.0],
"Long Context": [100.0, 99.0, 97.5, 95.0, 92.0],
},
}
The task breakdown reveals an important pattern. For general language modeling, all variants perform similarly because the task primarily requires local pattern matching. Reasoning tasks show more separation because they require the model to hold multiple pieces of evidence in mind simultaneously. Long-context tasks magnify this effect, as the model must retrieve and combine information from distant positions.
GQA-8 provides the best balance: within 1% of MHA on all tasks while giving substantial memory savings. This explains its adoption in LLaMA 2, Mistral, and other recent models.