Multi-Head Attention › Key Parameters

When configuring multi-head attention, the following parameters have the most significant impact on model behavior:

• num_heads (h): The number of parallel attention heads. Common values range from 8 to 96 depending on model size. More heads enable greater specialization but require the model dimension to be evenly divisible. Start with 8 heads for smaller models and scale proportionally with model dimension.

• d_model: The model dimension, which determines the total representation capacity. Must be divisible by num_heads. Standard values include 512 (original transformer), 768 (BERT-base), 1024 (BERT-large), and 12288 (GPT-3).

• d_k (head dimension): The dimension per head, computed as d_model // num_heads. Values between 64 and 128 work well in practice. Smaller values allow more heads but limit each head's expressiveness.

• d_v (value dimension): Often set equal to d_k. Can be set independently if you want asymmetric query-key versus value dimensions, though this is uncommon.

• Initialization scale: Projection matrices are typically initialized with variance $\frac{2}{d_{\text{model}}}$ or similar schemes. Poor initialization can cause attention weights to become too uniform or too peaked early in training.