Scaled Dot-Product Attention › Dot Product Similarity

The first step in scaled dot-product attention is measuring how relevant each key is to each query. We need a similarity function that is fast to compute, works well in high dimensions, and integrates naturally with gradient-based learning. The choice of similarity measure is not obvious, and it is worth understanding why the dot product was selected over alternatives.

One alternative is Euclidean distance: we could measure how close two vectors are in space and use closeness as a proxy for relevance. But Euclidean distance is expensive to compute for all $n^2$ query-key pairs, and minimizing distance does not naturally integrate with the maximize-gradient training objective. Another alternative is cosine similarity, which normalizes vectors to unit length before taking the dot product. Cosine similarity is interpretable, but the normalization step is expensive and, more importantly, the model cannot learn to increase similarity by scaling the magnitude of its projections, which turns out to be a useful degree of freedom.

The dot product satisfies all three requirements: it is fast to compute via matrix multiplication, its geometric interpretation aligns naturally with learned representations, and it allows the model to control similarity through both the angle and the magnitude of projection vectors. For two vectors $\mathbf{q}$ and $\mathbf{k}$ of dimension $d_k$, the dot product is:

$$\mathbf{q} \cdot \mathbf{k} = \sum_{l=1}^{d_k} q_l k_l$$

where:

• $q_l$: the $l$-th component of the query vector $\mathbf{q}$
• $k_l$: the $l$-th component of the key vector $\mathbf{k}$
• $d_k$: the shared dimension of queries and keys

Geometrically, the dot product measures how much two vectors point in the same direction. If $\mathbf{q}$ and $\mathbf{k}$ are unit vectors, the dot product equals their cosine similarity. For general vectors, it equals $\|\mathbf{q}\| \|\mathbf{k}\| \cos\theta$, where $\theta$ is the angle between them. Large positive values mean strong alignment; values near zero mean orthogonality; negative values mean the vectors point in opposite directions.

This geometric interpretation makes the dot product a natural fit for attention. During training, the model learns to align queries with relevant keys in the projected space. A position attending to relevant content will have its query vector pointing in a direction similar to the key vectors of those relevant positions. The projection matrices $W^Q$ and $W^K$ are jointly trained to produce this alignment: $W^Q$ shapes queries so they point toward relevant key directions, and $W^K$ shapes keys so they point toward the directions of queries that should attend to them. The learning process carves out a shared geometric space where relevance is encoded as directional alignment.

Why does this formula make sense? Notice that the dot product is bilinear: it is linear in $\mathbf{q}$ holding $\mathbf{k}$ fixed, and linear in $\mathbf{k}$ holding $\mathbf{q}$ fixed. This bilinearity is what makes the gradient flow back through both queries and keys cleanly during backpropagation. It is also what allows the score matrix to be computed as a single matrix multiplication, which modern hardware executes at peak efficiency.

In practice, we want to compute similarities between all $n$ queries and all $n$ keys simultaneously. Processing them one pair at a time would require $n^2$ separate dot product computations, which is slow and wasteful. Instead, we exploit the fact that all pairwise dot products can be expressed as a single matrix multiplication. This is a perfect job for linear algebra: the score matrix is:

$$S = QK^T$$

where:

• $Q \in \mathbb{R}^{n \times d_k}$: the query matrix, each row is one query vector
• $K^T \in \mathbb{R}^{d_k \times n}$: the transposed key matrix, each column is one key vector
• $S \in \mathbb{R}^{n \times n}$: the raw score matrix, where $S_{ij} = \mathbf{q}_i \cdot \mathbf{k}_j$

Entry $S_{ij}$ tells us how much position $i$'s query matches position $j$'s key. The entire matrix of $n^2$ scores is computed with a single matrix multiplication, which modern hardware executes very efficiently using BLAS routines that achieve near-peak floating-point throughput. This is one of the key reasons dot-product attention is so fast in practice: the score computation maps directly onto hardware-optimized matrix multiply.

The shape of $S$ also matters. For a sequence of $n$ tokens, $S$ has $n^2$ entries. Entry $(i, j)$ captures the directional alignment between token $i$'s query and token $j$'s key. The diagonal entries $S_{ii}$ capture each token's self-similarity, while off-diagonal entries capture cross-token dependencies. Before any normalization, these are called "logits" (by analogy with classification logits): raw unnormalized scores that will be converted into probabilities by softmax.