Efficiency Frontiers › Key Parameters
Michael BrenndoerferMarch 16, 20261 min read
Language AI HandbookAll chapters Part 67 of 67Frontier Outlook from 2025
Efficiency Frontiers › Key ParametersOne section · Sections in this chapter7 / 10 ↓
Sections in this chapter10
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The key parameters for the efficiency techniques covered in this chapter are:
- Quantization bits: Controls the tradeoff between precision and memory. 8-bit is nearly lossless; 4-bit requires careful calibration; 2-bit degrades quality significantly.
- KV heads (GQA): Number of key-value groups. Full MHA uses heads; MQA uses 1; GQA uses an intermediate value (4-8 is common). Reduces KV-cache memory proportionally.
- Speculative decoding k: Number of draft tokens per round. Larger increases potential speedup but also wastes more compute when tokens are rejected.
- Gradient checkpointing interval: Layers between saved checkpoints. Checkpointing every layers is theoretically optimal; in practice, checkpointing every layer or every other layer is common.
- Expert count and top-k (MoE): Total experts and how many are active per token. Common choices are 8 experts with top-2 or 64 experts with top-2. Increasing experts increases total capacity without increasing active FLOPs.
- FlashAttention tile size: Determined automatically based on SRAM size; controlled implicitly by block size hyperparameter in FlashAttention implementations.