Query, Key, Value › Key Parameters

When implementing QKV self-attention, these parameters control the mechanism's capacity and computational cost:

• embed_dim (d): The dimension of input token embeddings. This determines the input size to the projection matrices. Common values range from 256 to 4096 in production transformers.

• d_k (query/key dimension): The dimension of query and key vectors after projection. Smaller values reduce computation (since attention scores require $O(n^2 \cdot d_k)$ operations) but limit the expressiveness of query-key matching. Typically set to embed_dim / num_heads in multi-head attention.

• d_v (value dimension): The dimension of value vectors after projection. This determines the output dimension of the attention layer. Usually equals d_k for simplicity, but can differ when the output needs a different size than the matching space.

• Initialization scale: The projection matrices are typically initialized with small random values. Xavier/Glorot initialization scales weights by $\sqrt{2/(d_{\text{in}} + d_{\text{out}})}$, where $d_{\text{in}}$ is the input dimension and $d_{\text{out}}$ is the output dimension. This scaling helps maintain stable gradient magnitudes during training by keeping the variance of activations approximately constant across layers.

The ratio of d_k to embed_dim represents a compression factor. Setting d_k < embed_dim forces the model to learn a compressed representation for matching, which can act as a form of regularization. Setting d_k = embed_dim preserves full expressiveness but increases memory and computation.