Scaled Dot-Product Attention › Key Parameters

Scaled dot-product attention introduces two primary hyperparameters, both of which interact with the quadratic complexity we analyzed above.

The first is $d_k$ (query/key dimension): the dimensionality of the query and key projections. Larger $d_k$ allows queries and keys to encode more complex patterns. This provides more expressive capacity for the similarity computation. However, it increases computational cost as $O(n^2 d_k)$ and also increases the number of parameters in $W^Q$ and $W^K$. Too small a $d_k$ may not provide enough expressive capacity to represent the diversity of relationships the model needs to capture. Too large a $d_k$ wastes computation on dimensions that carry little signal. Typical values range from 64 to 256 per head in multi-head attention, with 64 being the value used in the original transformer paper's base model.

The second is $d_v$ (value dimension): the dimensionality of the value projections and the output. The value dimension controls the richness of the information each token contributes when attended to. Setting $d_v$ smaller than $d_k$ creates a bottleneck that forces the model to compress information, which can be a useful regularizer. Setting $d_v$ larger allows richer information transfer. In practice, $d_v = d_k$ is by far the most common choice, and it simplifies the architecture by making all projection dimensions equal. Changing $d_v$ changes the size of the output representation without affecting the attention weights.

The scaling factor $\sqrt{d_k}$ is not a hyperparameter: it is derived from $d_k$ and applied without modification. There is no reason to tune it separately; it is a fixed normalization constant whose derivation we worked through in the scaling section above.