Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Grouped Query Attention › Models Using GQA

Michael BrenndoerferUpdated August 6, 20252 min read

Language AI HandbookAll chapters

Part 29 of 67Modern Decoder Models

Chapter 209Grouped Query Attention › Models Using GQA

Reading section 9 of 14 · Models Using GQA · 2 minSection 9 of 14 · 2 min

Section 9 of 14 · 2 min read

Grouped Query Attention › Models Using GQAOne section · Sections in this chapter9 / 14 ↓

Sections in this chapter14

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

GQA has become the de facto standard for modern large language models. Here's how major models configure their attention:

In[22]:

Code

production_models = [
    {
        "name": "LLaMA 2 7B",
        "query_heads": 32,
        "kv_heads": 32,
        "attention": "MHA",
        "head_dim": 128,
    },
    {
        "name": "LLaMA 2 70B",
        "query_heads": 64,
        "kv_heads": 8,
        "attention": "GQA-8",
        "head_dim": 128,
    },
    {
        "name": "Mistral 7B",
        "query_heads": 32,
        "kv_heads": 8,
        "attention": "GQA-8",
        "head_dim": 128,
    },
    {
        "name": "Mixtral 8x7B",
        "query_heads": 32,
        "kv_heads": 8,
        "attention": "GQA-8",
        "head_dim": 128,
    },
    {
        "name": "Falcon 40B",
        "query_heads": 64,
        "kv_heads": 1,
        "attention": "MQA",
        "head_dim": 64,
    },
    {
        "name": "Falcon 180B",
        "query_heads": 64,
        "kv_heads": 8,
        "attention": "GQA-8",
        "head_dim": 128,
    },
    {
        "name": "Qwen 72B",
        "query_heads": 64,
        "kv_heads": 8,
        "attention": "GQA-8",
        "head_dim": 128,
    },
]

Out[23]:

Console

GQA in Production Models
======================================================================

Model              Attention    Q Heads    KV Heads   Ratio
----------------------------------------------------------------------
LLaMA 2 7B         MHA          32         32         1:1
LLaMA 2 70B        GQA-8        64         8          8:1
Mistral 7B         GQA-8        32         8          4:1
Mixtral 8x7B       GQA-8        32         8          4:1
Falcon 40B         MQA          64         1          64:1
Falcon 180B        GQA-8        64         8          8:1
Qwen 72B           GQA-8        64         8          8:1

Key observations:
- Smaller models (7B) often use MHA since memory isn't necessary
- Larger models (70B+) consistently use GQA-8
- GQA-8 has emerged as the industry standard ratio

A clear pattern emerges: smaller models often retain MHA because memory pressure is manageable, while models at 70B+ scale consistently adopt GQA with 8 key-value heads. This convergence suggests that 8 KV heads represent an empirically validated sweet spot for balancing quality and efficiency at scale.