Grouped Query Attention › Key Parameters
Language AI HandbookAll chapters Grouped Query Attention › Key ParametersOne section · Sections in this chapter12 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
When implementing or configuring GQA, the following parameters have the most significant impact on behavior:
-
num_heads(h): The number of query heads. Determines query diversity and must be divisible bynum_kv_heads. Common values: 32 (7B models), 64 (70B models). More heads enable more diverse attention patterns but require proportionally more query projection parameters. -
num_kv_heads(g): The number of key-value groups. Controls the memory-quality trade-off. Must dividenum_headsevenly. Common values: 8 for large models (GQA-8), 1 for MQA, equal tonum_headsfor MHA. Lower values reduce KV cache memory but may impact quality on complex reasoning tasks. -
head_dim(d_k): Dimension per attention head, computed asd_model // num_heads. Typical values: 64 or 128. Affects the expressiveness of each attention head and determines the KV cache size per head. -
d_model: Model dimension, must be divisible bynum_heads. Standard values: 4096 (7B), 8192 (70B). Larger dimensions increase model capacity but also increase memory requirements. -
Query-to-KV ratio (
num_heads // num_kv_heads): Determines how many query heads share each KV head. A ratio of 8:1 (GQA-8) has emerged as the industry standard. This provides 8x memory reduction with minimal quality impact.