Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Mistral Architecture › Grouped Query Attention

Michael BrenndoerferUpdated August 6, 20253 min read

Language AI HandbookAll chapters

Part 29 of 67Modern Decoder Models

Chapter 211Mistral Architecture › Grouped Query Attention

Reading section 5 of 14 · Grouped Query Attention · 3 minSection 5 of 14 · 3 min

Section 5 of 14 · 3 min read

Mistral Architecture › Grouped Query AttentionOne section · Sections in this chapter5 / 14 ↓

Sections in this chapter14

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Mistral employs grouped query attention (GQA), which reduces the number of key-value heads relative to query heads. Instead of the standard multi-head attention where each query head has its own key-value pair, GQA groups multiple query heads to share the same key-value heads.

Grouped Query Attention

GQA is a memory-bandwidth optimization where $h_q$ query heads share $h_{kv}$ key-value heads, with $h_q$ being a multiple of $h_{kv}$ . In this notation:

$h_q$ : number of query heads (32 for Mistral)
$h_{kv}$ : number of key-value heads (8 for Mistral)
$h_q / h_{kv}$ : the grouping ratio (4 for Mistral, meaning 4 query heads share each KV head)

This reduces KV cache size by a factor of $h_q / h_{kv}$ while maintaining most of the representational capacity of full multi-head attention.

For Mistral 7B:

Query heads: 32
KV heads: 8
Ratio: 4 query heads share each KV head

This means the KV cache is 4x smaller than it would be with standard multi-head attention, directly improving inference throughput by reducing memory bandwidth requirements.

Out[18]:

Visualization

Diagram showing 32 query heads grouped into 8 groups of 4, with each group sharing one KV head pair. — Grouped Query Attention in Mistral. Each group of 4 query heads (shades of blue) shares a single key-value head pair (orange/green). This reduces KV cache size by 4x while maintaining 32 independent query heads for representation.

The combination of GQA and the rolling buffer creates substantial memory savings. With 8 KV heads instead of 32, and a fixed window size instead of growing with sequence length, Mistral's inference memory footprint remains manageable even for long-context applications.