Architecture Hyperparameters › Key Parameters

When designing or analyzing transformer architectures, the following parameters have the most significant impact:

• $L$ (number of layers): Controls model depth and hierarchical abstraction capability. Start with 12-24 layers for smaller models (< 1B parameters) and scale to 32-80+ for larger models. Deeper models generally improve reasoning and multi-step tasks but require more careful initialization and training stability measures.

• $d_{\text{model}}$ (model dimension): The primary driver of per-layer capacity. Should be a multiple of 128 for optimal hardware utilization. Common values range from 768 (small models) to 8192 (frontier models). Quadratic impact on parameters means width is expensive.

• $n_{\text{heads}}$ (attention heads): Determines how many parallel attention patterns the model can learn. Set based on desired $d_{\text{head}}$: for $d_{\text{head}} = 64$, use $n_{\text{heads}} = d_{\text{model}} / 64$. More heads allow finer-grained attention but increase overhead.

• $d_{\text{head}}$ (head dimension): Target 64-128 for most applications. Smaller values allow more heads but reduce per-head expressiveness. Larger values (256+) rarely improve performance and reduce head count.

• $d_{\text{ff}}$ (FFN dimension): Set to $4 \times d_{\text{model}}$ for standard FFN or approximately $2.7 \times d_{\text{model}}$ for SwiGLU. Round to multiples of 256 for efficient GPU computation. This parameter dominates total parameter count.

• $n_{\text{kv\_heads}}$ (GQA heads): For inference-optimized models, reduce KV heads to 8-16 regardless of query head count. Reduces KV cache memory proportionally with minimal quality loss. Essential for long-context models.

• $V$ (vocabulary size): Typically 32K-100K tokens. Larger vocabularies improve compression but increase embedding memory. Choose based on target languages and tokenization strategy.