Scaled Dot-Product Attention: The Core Transformer Mechanism

In the previous chapter, we built the Query, Key, Value framework and saw how three learned projections give each token distinct representations for asking questions, advertising capabilities, and contributing information. We established the formula $Q = XW^Q$, $K = XW^K$, $V = XW^V$ and understood why separate roles matter. What we haven't yet addressed is: given these three matrices, how exactly do we compute the attention weights and the output? That's the job of scaled dot-product attention, the operation sitting at the heart of every transformer.

Scaled dot-product attention is deceptively simple. The complete formula is:

Four operations: a matrix multiply to compute similarities, a division to prevent numerical issues, a softmax to turn scores into weights, and a final matrix multiply to produce the output. Each piece solves a specific problem. Understanding those problems and their solutions is the goal of this chapter.

Think of scaled dot-product attention as a soft, differentiable database lookup. In a traditional database, you submit a query and get back exactly one record that matches. In attention, you submit a query and get back a weighted blend of all records, where the blend is proportional to how closely each record's key matches your query. The "softness" is what makes the whole system trainable: weights are smooth functions of the inputs, so gradients flow back through every attention weight to every parameter that influenced it.

the four operations in the formula are not arbitrary choices. Each one addresses a specific failure mode that would otherwise prevent the mechanism from training or working correctly. The dot product gives us a fast, geometrically meaningful similarity measure. The square-root scaling prevents the similarity values from growing so large that gradients vanish. The softmax converts raw similarities into a proper probability distribution. This ensures the output is a well-defined weighted average. And the matrix multiply with values efficiently aggregates information across the entire sequence. Strip out any one of these pieces and you either lose expressive power or make the system untrainable.

This chapter dissects each of those four pieces in detail, explains the mathematical reasons they work, and shows how they combine into the complete mechanism used by GPT, BERT, T5, and virtually every other modern language model. We also cover the two types of masking that adapt the same formula for different tasks: causal masking for autoregressive text generation, and padding masking for batched processing of variable-length sequences. By the end, you'll have a precise mental model of what happens inside every transformer attention layer, grounded in both intuition and the mathematics behind each design decision.

The first step in scaled dot-product attention is measuring how relevant each key is to each query. We need a similarity function that is fast to compute, works well in high dimensions, and integrates naturally with gradient-based learning. The choice of similarity measure is not obvious, and it is worth understanding why the dot product was selected over alternatives.

One alternative is Euclidean distance: we could measure how close two vectors are in space and use closeness as a proxy for relevance. But Euclidean distance is expensive to compute for all $n^2$ query-key pairs, and minimizing distance does not naturally integrate with the maximize-gradient training objective. Another alternative is cosine similarity, which normalizes vectors to unit length before taking the dot product. Cosine similarity is interpretable, but the normalization step is expensive and, more importantly, the model cannot learn to increase similarity by scaling the magnitude of its projections, which turns out to be a useful degree of freedom.

The dot product satisfies all three requirements: it is fast to compute via matrix multiplication, its geometric interpretation aligns naturally with learned representations, and it allows the model to control similarity through both the angle and the magnitude of projection vectors. For two vectors $\mathbf{q}$ and $\mathbf{k}$ of dimension $d_k$, the dot product is:

where:

• $q_l$: the $l$-th component of the query vector $\mathbf{q}$
• $k_l$: the $l$-th component of the key vector $\mathbf{k}$
• $d_k$: the shared dimension of queries and keys

Geometrically, the dot product measures how much two vectors point in the same direction. If $\mathbf{q}$ and $\mathbf{k}$ are unit vectors, the dot product equals their cosine similarity. For general vectors, it equals $\|\mathbf{q}\| \|\mathbf{k}\| \cos\theta$, where $\theta$ is the angle between them. Large positive values mean strong alignment; values near zero mean orthogonality; negative values mean the vectors point in opposite directions.

This geometric interpretation makes the dot product a natural fit for attention. During training, the model learns to align queries with relevant keys in the projected space. A position attending to relevant content will have its query vector pointing in a direction similar to the key vectors of those relevant positions. The projection matrices $W^Q$ and $W^K$ are jointly trained to produce this alignment: $W^Q$ shapes queries so they point toward relevant key directions, and $W^K$ shapes keys so they point toward the directions of queries that should attend to them. The learning process carves out a shared geometric space where relevance is encoded as directional alignment.

Why does this formula make sense? Notice that the dot product is bilinear: it is linear in $\mathbf{q}$ holding $\mathbf{k}$ fixed, and linear in $\mathbf{k}$ holding $\mathbf{q}$ fixed. This bilinearity is what makes the gradient flow back through both queries and keys cleanly during backpropagation. It is also what allows the score matrix to be computed as a single matrix multiplication, which modern hardware executes at peak efficiency.

In practice, we want to compute similarities between all $n$ queries and all $n$ keys simultaneously. Processing them one pair at a time would require $n^2$ separate dot product computations, which is slow and wasteful. Instead, we exploit the fact that all pairwise dot products can be expressed as a single matrix multiplication. This is a perfect job for linear algebra: the score matrix is:

where:

• $Q \in \mathbb{R}^{n \times d_k}$: the query matrix, each row is one query vector
• $K^T \in \mathbb{R}^{d_k \times n}$: the transposed key matrix, each column is one key vector
• $S \in \mathbb{R}^{n \times n}$: the raw score matrix, where $S_{ij} = \mathbf{q}_i \cdot \mathbf{k}_j$

Entry $S_{ij}$ tells us how much position $i$'s query matches position $j$'s key. The entire matrix of $n^2$ scores is computed with a single matrix multiplication, which modern hardware executes very efficiently using BLAS routines that achieve near-peak floating-point throughput. This is one of the key reasons dot-product attention is so fast in practice: the score computation maps directly onto hardware-optimized matrix multiply.

The shape of $S$ also matters. For a sequence of $n$ tokens, $S$ has $n^2$ entries. Entry $(i, j)$ captures the directional alignment between token $i$'s query and token $j$'s key. The diagonal entries $S_{ii}$ capture each token's self-similarity, while off-diagonal entries capture cross-token dependencies. Before any normalization, these are called "logits" (by analogy with classification logits): raw unnormalized scores that will be converted into probabilities by softmax.

Each row of the score matrix corresponds to one token's query matched against all keys. Notice that the scores span a wide range and include negative values. These raw scores aren't attention weights yet; we need two more transformations before they are usable. In particular, the values here are on a scale where some entries are several times larger than others, which would cause softmax to produce extremely uneven weight distributions. Before we apply softmax, we need to address the scale problem.

Before applying softmax to the score matrix, we must address a subtle but critical issue: as $d_k$ grows larger, the dot products tend to grow very large in magnitude, and softmax applied to large values has an extremely undesirable property. This is not a hypothetical edge case: it is a systematic failure mode that the original transformer paper identified and explicitly corrected with the $\sqrt{d_k}$ scaling factor. Without this fix, training transformers with large key dimensions would be dramatically slower and potentially unstable.

To see why dot products grow with dimension, assume that the components of $\mathbf{q}$ and $\mathbf{k}$ are independent random variables with mean 0 and variance 1. This is a reasonable initialization assumption for learned projection vectors. Then each term $q_l k_l$ has mean 0 and variance 1, since the product of two independent mean-zero unit-variance variables has variance equal to the product of their variances. The dot product is a sum of $d_k$ such terms:

By properties of variance, the variance of this sum is $d_k$, making the standard deviation $\sqrt{d_k}$. So as $d_k$ grows, dot products have a spread proportional to $\sqrt{d_k}$. A $d_k$ of 64 yields dot products with standard deviation around 8. A $d_k$ of 512 yields dot products with standard deviation around 22.6. These are large values to feed into softmax.

Now consider what softmax does to large inputs. The softmax converts a vector of real numbers into a probability distribution. Given an input vector $\mathbf{z}$ with $n$ elements, the softmax output for element $i$ is:

where:

• $z_i$: the $i$-th score (the raw dot product value for one query-key pair)
• $e^{z_i}$: the exponential of that score, always positive
• $\sum_j e^{z_j}$: the sum of exponentials over all positions, serving as a normalizing constant

When one value in $\mathbf{z}$ is substantially larger than the others, the exponential amplifies the gap dramatically. For example, if $\mathbf{z} = [0, 0, 10]$, the softmax is approximately $[0.0000226, 0.0000226, 0.9999548]$: a near-one-hot vector that places almost all attention on the third position. Gradients through softmax in this regime are nearly zero everywhere except at the maximum, which means parameters receive almost no learning signal. The exponential function's property of amplifying differences between values is exactly what creates this problem: a difference of 10 units in the input becomes a factor of $e^{10} \approx 22000$ in the numerators, completely overwhelming all but the largest value.

This is the saturation problem: large inputs cause softmax to produce extremely peaked distributions, which kills gradient flow and makes training difficult. Think of it like a light switch stuck in the on position: the model collapses to always attending to one position, gradients cannot push it to distribute attention more broadly, and the mechanism loses its capacity to capture dependencies across multiple positions.

The original "Attention Is All You Need" paper introduced a simple fix: divide the dot products by $\sqrt{d_k}$ before applying softmax. The purpose of this step is to counteract the variance growth we identified above. The scaled score matrix becomes:

where:

• $QK^T \in \mathbb{R}^{n \times n}$: the raw score matrix from matrix multiplication
• $\sqrt{d_k}$: the scaling factor, the square root of the query/key dimension
• $\tilde{S} \in \mathbb{R}^{n \times n}$: the scaled scores, ready for softmax

Why $\sqrt{d_k}$ specifically? Dividing by $\sqrt{d_k}$ normalizes the variance. Recall that the raw dot products have standard deviation $\sqrt{d_k}$ under our initialization assumption. After dividing by $\sqrt{d_k}$, the standard deviation becomes 1, just like unit-variance inputs. This keeps the softmax operating in its sensitive, gradient-rich regime regardless of how large $d_k$ is. You can think of it as a standardization step: instead of letting the scale of the inputs to softmax depend on an architectural hyperparameter, we normalize them so the effective scale is always around 1.

Why does this formula make sense? Notice that scaling by $1/\sqrt{d_k}$ does not change the ranking of scores: if key $j$ had the highest raw score for query $i$, it still has the highest scaled score. What changes is the sharpness of the resulting softmax distribution. With variance-normalized inputs, softmax produces distributions in the range where gradients are non-trivial, and the model can learn to make them sharper or flatter as needed by adjusting the directions of its query and key vectors. The scaling constant is simply a normalization factor, not a learnable parameter.

After scaling, the scores have a standard deviation close to 1, placing softmax in its operating range. The scaling factor is a constant; it doesn't change what the model learns, only the numerical scale of the inputs to softmax. This is a good example of a design pattern common in deep learning: a simple normalization step that costs almost nothing computationally but enables significantly more stable and effective training.

With scaled scores in hand, we apply softmax row-wise to convert each token's scores into a probability distribution over positions. The purpose of this step is to ensure the attention weights are non-negative and sum to one, which is necessary for the subsequent weighted sum of values to be a proper convex combination. Without softmax, the weights could be negative or unbounded, making the output an arbitrary linear combination rather than a meaningful weighted average.

The softmax is applied independently to each row of the scaled score matrix:

where the softmax is applied to each row independently. Entry $A_{ij}$ gives the attention weight: the fraction of position $j$'s value that position $i$ incorporates into its output. The attention weights satisfy:

and all entries are non-negative ($A_{ij} \geq 0$), making each row a proper probability distribution.

The softmax has a critical secondary effect: it sharpens the distribution. Positions with high raw scores receive exponentially more attention than positions with low scores. After training, the model learns to assign high scores to relevant positions, producing peaked attention distributions that focus on key information while still being smooth enough to allow gradient-based learning. This sharpening effect is what gives attention its ability to be selective: a model that has learned to identify the antecedent of a pronoun will have a very peaked attention distribution when that pronoun is the current query, with nearly all weight on the antecedent token.

Softmax is applied row-wise, not to the entire matrix. Each query independently produces its own probability distribution over keys. This means different tokens can attend to entirely different parts of the sequence simultaneously, which is exactly the flexibility we need. Token $i$'s attention distribution is completely independent of token $j$'s, even though they are computed from the same key matrix.

The final step uses the attention weights to aggregate the value vectors. For each query position $i$, we compute a weighted combination of all value vectors. This step produces the enriched representations: it takes information from all positions in the sequence and blends it together, with the blend controlled by the attention weights we just computed.

To understand why this design makes sense, consider the alternative: using the keys directly for the output instead of having a separate value projection. If we used keys as values, the same representation would serve two incompatible purposes: advertising relevance (key role) and contributing information (value role). Keeping them separate gives the model the freedom to have token $j$'s key say "I am relevant to syntactic subject queries" while its value says "here is the rich semantic content I contribute when selected." This decoupling is what allows attention heads to specialize in different linguistic functions.

For each query position $i$, the output is:

where:

• $A_{ij}$: the attention weight from position $i$ to position $j$, giving the fraction of position $j$'s value to include
• $\mathbf{v}_j$: the value vector for position $j$, containing the information that position $j$ contributes when selected
• $\text{output}_i$: the attention output for position $i$, a $d_v$-dimensional context-enriched vector

In matrix form across all positions simultaneously:

where:

• $A \in \mathbb{R}^{n \times n}$: the attention weight matrix, rows are probability distributions over positions
• $V \in \mathbb{R}^{n \times d_v}$: the value matrix, rows are value vectors for each position
• $AV \in \mathbb{R}^{n \times d_v}$: the output matrix, rows are attention outputs for each position

The output for each position is a soft lookup: a blended summary of information from all positions, weighted by relevance. Positions that received high attention weights contribute most; positions with low weights contribute little. The output has shape $(n, d_v)$, matching the input shape (with $d_v$ instead of the original embedding dimension $d$). This means each token gets a new representation of the same dimension, enriched by context from the whole sequence.

Why is this better than a hard lookup (selecting the single most relevant position)? Because language rarely has crisp, discrete dependencies. Understanding "it" in "The cat sat on the mat and it purred" requires integrating information from multiple positions: "cat" is most relevant, but the verb tense and surrounding context also contribute. The soft weighted sum captures this gracefully. It is also differentiable everywhere, which means gradients flow smoothly through the output back to both the value vectors and the attention weights, allowing end-to-end training to optimize all components jointly.

Each row of the output is a $d_v$-dimensional vector: a context-aware representation of that token, blended from value vectors according to attention weights. Tokens that attended broadly have outputs that are smooth blends of many values; tokens that attended sharply to a few positions have outputs dominated by those positions' values. After this step, each token's representation contains information from the entire sequence, weighted by learned relevance, which is precisely what makes attention powerful for understanding language.

Putting it all together, the complete scaled dot-product attention formula is:

This single formula packages four sequential operations:

1. $QK^T$: compute all pairwise query-key similarities as a matrix multiplication, producing an $n \times n$ score matrix
2. $/ \sqrt{d_k}$: scale to prevent softmax saturation, normalizing variance to approximately 1
3. $\text{softmax}(\cdot)$: normalize each row into an attention weight distribution, producing non-negative weights summing to 1
4. $\times V$: produce context-enriched outputs as weighted sums of value vectors, blending information across positions

Each operation is differentiable everywhere (the scaling is a constant, softmax is smooth, and matrix multiply is linear), so gradients flow cleanly backward through the entire chain. This is what makes end-to-end training possible: a loss computed on the final outputs can propagate gradients back through the weighted sum, back through the softmax, back through the scaling and dot products, and ultimately back to the projection matrices $W^Q$, $W^K$, and $W^V$ that produced $Q$, $K$, and $V$.

This implementation handles the common case without masking. The optional mask argument will become important when we discuss causal attention in the next section. Notice that the function is entirely self-contained: given $Q$, $K$, and $V$, it computes the output in four lines of code. The elegance of the implementation reflects the mathematical elegance of the formula itself.

So far, attention has been bidirectional: each position can attend to every other position, including those that come later in the sequence. This is fine for encoding tasks like classification or feature extraction, where the full sequence is available at inference time. But for language generation, where we predict token $t+1$ given tokens $1$ through $t$, we need to prevent each position from seeing future tokens. This is causal (or autoregressive) attention.

The reason this constraint is critical goes beyond preventing "cheating." If a model is allowed to attend to future tokens during training, it learns to predict tokens by copying from the future rather than modeling language. The training loss would be low, but the model would be useless for generation, where the future tokens are exactly what we're trying to predict. Causal masking enforces the correct inductive bias: the model must learn to predict each token from the past context alone.

The mechanism is straightforward: before applying softmax, we add a very large negative value (effectively $-\infty$) to the scores for all positions that should be masked. After softmax, $e^{-\infty} = 0$, so those positions receive zero attention weight. The elegance of this approach is that it requires no changes to the softmax function itself: the masking is applied to the inputs of softmax, and softmax naturally produces zero weights for $-\infty$ inputs.

The causal mask is an upper-triangular matrix of True values (above the diagonal represents future positions):

where $i$ is the query position (row) and $j$ is the key position (column). Position $i$ can attend to positions $0, 1, \ldots, i$ but not to positions $i+1, i+2, \ldots, n-1$. The diagonal entries ($j = i$, self-attention) are not masked: each token can always attend to itself.

Beyond causal masking, transformers also use padding masks to handle variable-length sequences. When batching sequences of different lengths, shorter sequences are padded with a special token to reach a common length. The model should not attend to padding positions, since they contain no real information: a padded position is an artifact of the batching process, not a meaningful part of the input.

The problem with attending to padding positions is subtle but real. If a padding token happens to receive a high dot-product score, the softmax will assign it a non-trivial weight, and the padding token's value vector will contribute to the output. Since padding tokens carry no semantic content, this is wasted attention capacity at best and harmful noise at worst. The padding mask removes this possibility by zeroing out attention weights for all padding positions.

A padding mask marks padded positions as invalid by setting the corresponding columns of the score matrix to $-\infty$ before softmax, just as the causal mask does for future positions. Combined with the causal mask (when needed), the model correctly ignores both future tokens and padding tokens.

The first three tokens attend only to earlier (non-padded) tokens within their causal window. Tokens 3 and 4 are padding positions; in practice we discard their outputs, but their attention patterns correctly reflect only valid positions. Notice that combining masks is simple: we take the logical OR of the causal mask and the padding mask, and any position that is masked by either constraint is masked in the combined mask.

In practice, transformers process batches of sequences simultaneously. Training on a single sequence at a time would leave GPUs severely underutilized: modern GPUs achieve peak throughput when they can process hundreds or thousands of examples in parallel. The attention formula naturally extends to batched computation by adding a batch dimension. For a batch of $B$ sequences each of length $n$:

• $Q \in \mathbb{R}^{B \times n \times d_k}$: query matrices for all sequences in the batch
• $K \in \mathbb{R}^{B \times n \times d_k}$: key matrices for all sequences in the batch
• $V \in \mathbb{R}^{B \times n \times d_v}$: value matrices for all sequences in the batch
• Output $\in \mathbb{R}^{B \times n \times d_v}$: enriched representations for all sequences

The matrix multiplication $QK^T$ uses batched matrix multiply: for each example in the batch, we compute the $n \times n$ score matrix independently. NumPy and PyTorch handle this efficiently through the @ operator or torch.bmm. The key point is that sequences in a batch are completely independent: the attention weights for sequence 1 do not influence the attention weights for sequence 2. The batch dimension is purely an efficiency mechanism, not a semantic one.

The batched formulation is identical mathematically to processing each example independently, but the vectorized computation is substantially faster on GPUs due to their parallel execution model. On a modern GPU, processing a batch of 32 sequences takes roughly the same wall-clock time as processing a single sequence, because all 32 matrix multiplications can execute in parallel across thousands of GPU cores.

The $\sqrt{d_k}$ scaling prevents saturation at the start of training. It is also essential for maintaining numerical stability throughout training as model weights adapt. This distinction is worth dwelling on, because it explains why the scaling is a permanent architectural feature rather than just an initialization trick.

Consider what can go wrong without scaling. As the model trains, projection matrices $W^Q$ and $W^K$ can grow in magnitude to produce peaked attention distributions. Gradient descent does not naturally constrain the scale of learned parameters: if slightly larger weights consistently reduce the loss, the optimizer will keep increasing them. Once queries and keys have large dot products, softmax saturates, gradients vanish, and parameters stop updating. The model gets stuck with sharp, nearly-fixed attention patterns that cannot adapt as training proceeds. The training curve flattens, and the model is trapped in a suboptimal configuration.

With $\sqrt{d_k}$ scaling, the effective inputs to softmax remain in a numerically stable range. The model can still learn to produce peaked attention (by making queries and keys point in similar directions), but the gradients remain non-zero throughout, enabling continued learning. Think of the scaling as maintaining an implicit contract: the softmax always receives inputs with unit-scale variance, regardless of how the learned weights have grown. This contract is what keeps training stable over thousands of gradient updates.

There is a secondary numerical consideration: the softmax computation itself. A naive implementation computes $e^{z_i}$ for each score $z_i$. Even after scaling, some scores may be positive enough that $e^{z_i}$ overflows floating-point representation. The standard fix is the log-sum-exp trick: subtract the maximum score before exponentiating.

where:

• $z_i$: the $i$-th score value
• $z_{\max} = \max_j z_j$: the maximum score value in the row
• $e^{z_i - z_{\max}}$: the shifted exponential, at most $e^0 = 1$, preventing overflow
• $\sum_j e^{z_j - z_{\max}}$: the shifted normalization constant

Subtracting the max shifts the largest value to zero, preventing overflow while producing identical results mathematically (the max cancels in both numerator and denominator). Why does this formula make sense? Notice that $\frac{e^{z_i - z_{\max}}}{\sum_j e^{z_j - z_{\max}}} = \frac{e^{z_i} / e^{z_{\max}}}{\sum_j e^{z_j} / e^{z_{\max}}} = \frac{e^{z_i}}{\sum_j e^{z_j}}$, which is exactly the original softmax. The shift is mathematically neutral but numerically essential.

Our softmax function already uses the log-sum-exp trick. Production implementations in PyTorch and JAX also use this by default, so overflow is typically not a concern in practice, but understanding why it matters explains why the stable version is the standard. When building your own attention implementations, always use the numerically stable version.

Scaled dot-product attention has a computational cost that shapes how transformers scale with sequence length. Understanding this cost motivates both current architectural choices and research into more efficient attention mechanisms. The quadratic scaling is the primary bottleneck limiting the sequence length modern transformers can process.

The dominant cost is the $QK^T$ matrix multiplication. Multiplying an $n \times d_k$ matrix by a $d_k \times n$ matrix requires $O(n^2 d_k)$ multiply-add operations. The subsequent $AV$ multiply is $O(n^2 d_v)$. Together, the total computation is:

where $d = \max(d_k, d_v)$, assuming $d_k$ and $d_v$ are comparable. The quadratic factor $n^2$ means that doubling the sequence length quadruples the computation, and multiplying the sequence length by 10 increases computation by 100x.

The memory cost is also $O(n^2)$: we must store the $n \times n$ score matrix and the $n \times n$ attention weight matrix. For typical sequence lengths of 512 or 1024 tokens with 32-bit floats, this is manageable. But for very long sequences (tens of thousands of tokens), storing $O(n^2)$ matrices becomes infeasible. A sequence of length 32,768 tokens would require a single attention matrix with over a billion entries, far exceeding available GPU memory.

Theory becomes concrete when you trace through a small example by hand. Let's work through a minimal but complete attention computation with a 4-token sequence: "the", "cat", "sat", "there". We'll use small dimensions ($d_k = d_v = 3$) so every intermediate value fits on screen and we can interpret each step.

The goal of this example is to make the abstract operations tangible. When we say "query $i$ attends to key $j$," what does that look like numerically? When we say "the output is a weighted blend of values," what blend weights are we computing? Working through these questions with concrete numbers builds the intuition needed to understand attention in larger models.

Think of this worked example as running the attention formula in slow motion: we pause at each step to print the intermediate matrices, check their shapes, and interpret what the numbers mean before moving to the next step.

Walk through what happened at each step. The raw score matrix in Step 1 contains unscaled query-key dot products: each entry is the raw similarity between one token's query and one token's key, before any normalization. After scaling in Step 2, the values shrink by a factor of $\sqrt{3} \approx 1.73$, bringing them into the range where softmax operates without saturation. The attention weight matrix in Step 3 shows which positions each token attends to most: each row is a proper probability distribution, and you can see which keys received the most attention weight from each query. Finally, Step 4 shows the output for each token, which is a 3-dimensional vector representing that token's context-enriched representation after blending in information from across the sequence.

One observation worth making: even though the token sequence has semantic structure, these attention patterns are driven by random projection matrices rather than learned ones. In a trained model, the patterns would be far more meaningful: "the" might strongly attend to the noun it modifies ("cat"), "sat" might attend to the subject ("cat") and object ("there"), and so on. The random weights here demonstrate the mechanics; training is what makes the weights encode linguistic structure.

Scaled dot-product attention introduces two primary hyperparameters, both of which interact with the quadratic complexity we analyzed above.

The first is $d_k$ (query/key dimension): the dimensionality of the query and key projections. Larger $d_k$ allows queries and keys to encode more complex patterns. This provides more expressive capacity for the similarity computation. However, it increases computational cost as $O(n^2 d_k)$ and also increases the number of parameters in $W^Q$ and $W^K$. Too small a $d_k$ may not provide enough expressive capacity to represent the diversity of relationships the model needs to capture. Too large a $d_k$ wastes computation on dimensions that carry little signal. Typical values range from 64 to 256 per head in multi-head attention, with 64 being the value used in the original transformer paper's base model.

The second is $d_v$ (value dimension): the dimensionality of the value projections and the output. The value dimension controls the richness of the information each token contributes when attended to. Setting $d_v$ smaller than $d_k$ creates a bottleneck that forces the model to compress information, which can be a useful regularizer. Setting $d_v$ larger allows richer information transfer. In practice, $d_v = d_k$ is by far the most common choice, and it simplifies the architecture by making all projection dimensions equal. Changing $d_v$ changes the size of the output representation without affecting the attention weights.

The scaling factor $\sqrt{d_k}$ is not a hyperparameter: it is derived from $d_k$ and applied without modification. There is no reason to tune it separately; it is a fixed normalization constant whose derivation we worked through in the scaling section above.

Scaled dot-product attention is elegant and effective, but several limitations shape how it's used in practice. Understanding these limitations is not just academic: they directly motivate architectural choices in every major language model, and they explain why significant research effort has gone into attention variants over the past several years.

The most significant limitation is the $O(n^2)$ computational and memory cost. For a sequence of 1024 tokens processed with 12 attention heads, the attention matrices alone require storing millions of floating-point values per layer. Extending to 100,000 tokens (the range needed for document-level tasks such as analyzing entire research papers or legal contracts) makes standard attention impractical without modification. A sequence of 100,000 tokens would require $10^{10}$ entries per attention layer, far beyond the memory capacity of any current hardware. Research into efficient attention has produced many alternatives: sparse attention (attending to a fixed local window plus selected global tokens), low-rank approximations (approximating the $n \times n$ matrix as a product of smaller matrices), and kernel-based methods (reformulating attention to avoid materializing the full matrix). None has fully replaced standard attention for general-purpose models, but each has found niches where it offers meaningful advantages.

Softmax attention also has a tendency toward attention dilution on long sequences. Because softmax normalizes over all positions, each attention weight becomes smaller as sequence length grows: if the sequence has 1000 tokens, the average attention weight is 1/1000, even if the most relevant position truly deserves a weight of 0.5. This means the model must learn to produce very large raw scores to overcome the normalization and assign meaningful weight to relevant positions. In practice, this can cause trained models to show unexpectedly diffuse attention on long sequences, where they fail to focus on the most relevant context even when that context is clearly present. Researchers have explored alternatives like entmax (a sparse softmax that produces exactly zero weights for irrelevant positions) and attention temperature scaling at inference time, but softmax remains the dominant choice due to its well-understood behavior and hardware efficiency.

From a representational standpoint, standard scaled dot-product attention is permutation-equivariant: swapping two tokens in the input swaps the corresponding rows and columns of the attention matrix, and the output rows swap accordingly. The mechanism has no inherent notion of position or order. Feed the same sentence in different word orders, and the attention mechanism treats it as the same set of tokens in a different arrangement. This is why transformers require explicit positional encoding: without it, the model cannot distinguish "the cat sat on the mat" from "the mat sat on the cat." Positional encodings inject order information into the token representations before they enter the attention computation, effectively breaking the permutation equivariance. We'll cover positional encodings in Part XIV of the handbook.

There is also a more subtle limitation related to the form of the similarity function. Dot-product attention computes similarity as a bilinear form, which means the similarity score between query $i$ and key $j$ can only depend on the specific pair $(i, j)$ through their dot product. This is a restricted form of interaction: there is no way for the attention score to depend on the relationship between key $j$ and other keys, or on global properties of the sequence. More expressive attention variants have been proposed, but the dot product's simplicity and efficiency have kept it dominant.

Scaled dot-product attention maps QKV inputs into context-enriched representations through four sequential operations:

• Dot product ($QK^T$): measures pairwise similarity between all queries and all keys, producing an $n \times n$ score matrix with $O(n^2 d_k)$ cost
• Scaling ($/ \sqrt{d_k}$): normalizes scores to have unit variance, preventing softmax saturation and enabling stable gradient flow throughout training
• Softmax: converts scores row-wise into probability distributions over positions, producing attention weights $A_{ij} \geq 0$ with $\sum_j A_{ij} = 1$
• Weighted sum ($AV$): aggregates value vectors according to attention weights, producing a $d_v$-dimensional context-aware representation for each position

The causal mask extends this mechanism to autoregressive generation by zeroing out attention to future positions before softmax. Combined with the padding mask for batched computation, the masking mechanism handles all common production use cases. Both types of masking work by adding $-\infty$ to score matrix entries before softmax, which causes those positions to receive exactly zero weight in the output.

The quadratic complexity $O(n^2 d)$ in both computation and memory is the central scalability challenge. Efficient attention variants address this bottleneck, which we'll examine in the Attention Complexity chapter. Before that, the next chapter covers multi-head attention, which runs multiple instances of scaled dot-product attention in parallel to allow the model to attend to different aspects of the input simultaneously. Multi-head attention is the direct generalization of everything we've covered here: it applies scaled dot-product attention not once but $h$ times, with different learned projections for each head, and concatenates the results. Understanding scaled dot-product attention fully, as we've done in this chapter, is the necessary foundation for understanding why multi-head attention works so well in practice.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about scaled dot-product attention.