Multi-Query Attention › Quality Trade-offs

The extreme parameter sharing in MQA raises an important question: what do we lose by sharing keys and values across all heads? The answer depends on the task.

In standard MHA, each head can specialize its key and value representations. Research has shown that different heads often capture distinct linguistic phenomena:

• Syntactic heads: Attend to grammatical structure (subject-verb, modifier-noun)
• Semantic heads: Capture meaning relationships (synonymy, entailment)
• Positional heads: Focus on local context or specific positions
• Rare pattern heads: Specialize in infrequent but important patterns

When we share K and V, all heads must use the same "retrieval index" (keys) and the same "information content" (values). They can still differ in what they query for (different Q projections), but they're constrained to retrieve from the same representation.

The original MQA paper and subsequent studies found modest quality degradation. Let's examine representative benchmark results across translation and summarization benchmarks, as well as language-modeling tasks.

The quality degradation typically falls in the 1-3% range. For many applications, especially inference-heavy deployments where throughput matters more than the last percentage point of accuracy, this is an excellent trade-off.

Some tasks are more sensitive to the reduced expressiveness:

• Long-range reasoning: Tasks requiring diverse attention patterns over long contexts
• Multi-task learning: Models expected to handle many different tasks
• Fine-grained linguistic tasks: Parsing, detailed entity typing, relation extraction

For these applications, Grouped Query Attention (GQA) offers a middle ground, sharing K/V among groups of heads rather than all heads.