Multi-Query Attention › Key Parameters

When implementing or configuring MQA and GQA, these parameters have the greatest impact on the memory-quality trade-off:

• num_heads (int): The number of query heads. This determines model expressiveness and directly affects the cache reduction factor when using MQA. Typical values range from 32 (7B models) to 128 (largest models).
• num_kv_heads (int): The number of key-value heads. Set to 1 for MQA, equal to num_heads for standard MHA, or an intermediate value for GQA. Common GQA configurations use 4-8 KV heads regardless of query head count.
• head_dim (int): The dimension of each attention head. Standard values are 64 or 128. Larger head dimensions increase per-token cache size but may improve attention quality. Most modern LLMs use 128.
• hidden_size (int): The model's hidden dimension, typically equal to num_heads * head_dim. This determines the size of projection matrices and overall model capacity.

When choosing an attention variant, compare the full-cache MHA baseline with GQA and MQA against your deployment constraints. If memory is severely limited or batch sizes must be large, MQA provides maximum savings. If quality is the priority and memory is less constrained, GQA with 4-8 KV heads offers a balanced approach that most production systems favor.