Grouped Query Attention › Models Using GQA

Michael BrenndoerferUpdated August 6, 20252 min read
Language AI HandbookAll chapters
Part 29 of 67Modern Decoder Models
Chapter 209Grouped Query Attention › Models Using GQA
Reading section 9 of 14 · Models Using GQA · 2 minSection 9 of 14 · 2 min

Section 9 of 14 · 2 min read

Grouped Query Attention › Models Using GQASection view · Sections in this chapter9 / 14 ↓
Reading tools Definitions, links & notes

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Notes & comments

GQA has become the de facto standard for modern large language models. Here's how major models configure their attention:

PythonIn [22]
CollapseExpand
production_models = [
    {
        "name": "LLaMA 2 7B",
        "query_heads": 32,
        "kv_heads": 32,
        "attention": "MHA",
        "head_dim": 128,
    },
    {
        "name": "LLaMA 2 70B",
        "query_heads": 64,
        "kv_heads": 8,
        "attention": "GQA-8",
        "head_dim": 128,
    },
    {
        "name": "Mistral 7B",
        "query_heads": 32,
        "kv_heads": 8,
        "attention": "GQA-8",
        "head_dim": 128,
    },
    {
        "name": "Mixtral 8x7B",
        "query_heads": 32,
        "kv_heads": 8,
        "attention": "GQA-8",
        "head_dim": 128,
    },
    {
        "name": "Falcon 40B",
        "query_heads": 64,
        "kv_heads": 1,
        "attention": "MQA",
        "head_dim": 64,
    },
    {
        "name": "Falcon 180B",
        "query_heads": 64,
        "kv_heads": 8,
        "attention": "GQA-8",
        "head_dim": 128,
    },
    {
        "name": "Qwen 72B",
        "query_heads": 64,
        "kv_heads": 8,
        "attention": "GQA-8",
        "head_dim": 128,
    },
]
ConsoleOut [23]
CollapseExpand
GQA in Production Models
======================================================================

Model              Attention    Q Heads    KV Heads   Ratio
----------------------------------------------------------------------
LLaMA 2 7B         MHA          32         32         1:1
LLaMA 2 70B        GQA-8        64         8          8:1
Mistral 7B         GQA-8        32         8          4:1
Mixtral 8x7B       GQA-8        32         8          4:1
Falcon 40B         MQA          64         1          64:1
Falcon 180B        GQA-8        64         8          8:1
Qwen 72B           GQA-8        64         8          8:1

Key observations:
- Smaller models (7B) often use MHA since memory isn't necessary
- Larger models (70B+) consistently use GQA-8
- GQA-8 has emerged as the industry standard ratio

A clear pattern emerges: smaller models often retain MHA because memory pressure is manageable, while models at 70B+ scale consistently adopt GQA with 8 key-value heads. This convergence suggests that 8 KV heads represent an empirically validated sweet spot for balancing quality and efficiency at scale.

Comments

Add to the conversation

A question, a correction, or an example from your own work.

No comments yet. Be the first to share your thoughts!

Newsletter

Stay up to date

Get new articles, book updates, and notes on AI and technology.

Unsubscribe at any time. Privacy policy