Architecture Hyperparameters › Summary

Transformer architecture hyperparameters determine model capacity, computational requirements, and ultimately performance. The key takeaways from this chapter are:

• Six core hyperparameters define a transformer: vocabulary size ($V$), layers ($L$), model dimension ($d_{\text{model}}$), attention heads ($n_{\text{heads}}$), head dimension ($d_{\text{head}}$), and FFN dimension ($d_{\text{ff}}$). Everything else derives from these choices.

• Depth scales linearly, width quadratically. Doubling the number of layers doubles parameters; doubling the model dimension quadruples them. This makes depth a more parameter-efficient way to add capacity, but both contribute to model expressiveness in different ways.

• Head dimensions cluster around 64-128. This empirical sweet spot balances per-head capacity with the benefits of multiple parallel attention mechanisms. The number of heads is then determined by $n_{\text{heads}} = d_{\text{model}} / d_{\text{head}}$.

• FFN expansion follows predictable ratios. Standard FFNs use $d_{\text{ff}} = 4 \times d_{\text{model}}$, while SwiGLU-based FFNs use approximately $2.7 \times d_{\text{model}}$ to maintain equivalent parameter counts.

• Grouped-query attention trades parameters for memory. GQA reduces KV cache size during inference by sharing key-value projections across multiple query heads, enabling longer context windows without proportional memory increases.

• FFN dominates parameter counts. At larger scales, feed-forward networks contain the majority of parameters, making them the primary target for efficiency innovations like mixture-of-experts.

• Hardware alignment matters. Dimensions should be multiples of 8, 64, or 128 for efficient tensor core utilization. The number of heads should divide evenly across GPUs for tensor parallelism.

When designing a new architecture, start with proven configurations from similar-scale models, ensure hardware-friendly dimensions, and adjust based on your specific constraints and empirical results. The art of architecture design lies in balancing these considerations while respecting the fundamental trade-offs between capacity and compute, alongside memory.