Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Scaled Dot-Product Attention › The Full Attention Formula

Michael BrenndoerferMay 25, 20252 min read

Language AI HandbookAll chapters

Part 13 of 67Self-Attention

Chapter 92Scaled Dot-Product Attention › The Full Attention Formula

Reading section 6 of 15 · The Full Attention Formula · 2 minSection 6 of 15 · 2 min

Section 6 of 15 · 2 min read

Scaled Dot-Product Attention › The Full Attention FormulaOne section · Sections in this chapter6 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Putting it all together, the complete scaled dot-product attention formula is:

\text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{QK^T}{\sqrt{d_k}}\right) V

This single formula packages four sequential operations:

$QK^T$ : compute all pairwise query-key similarities as a matrix multiplication, producing an $n \times n$ score matrix
$/ \sqrt{d_k}$ : scale to prevent softmax saturation, normalizing variance to approximately 1
$\text{softmax}(\cdot)$ : normalize each row into an attention weight distribution, producing non-negative weights summing to 1
$\times V$ : produce context-enriched outputs as weighted sums of value vectors, blending information across positions

Each operation is differentiable everywhere (the scaling is a constant, softmax is smooth, and matrix multiply is linear), so gradients flow cleanly backward through the entire chain. This is what makes end-to-end training possible: a loss computed on the final outputs can propagate gradients back through the weighted sum, back through the softmax, back through the scaling and dot products, and ultimately back to the projection matrices $W^Q$ , $W^K$ , and $W^V$ that produced $Q$ , $K$ , and $V$ .

In[15]:

Code

def scaled_dot_product_attention(Q, K, V, mask=None):
    """
    Scaled dot-product attention.

    Args:
        Q: Query matrix, shape (n, d_k)
        K: Key matrix, shape (n, d_k)
        V: Value matrix, shape (n, d_v)
        mask: Optional boolean mask, shape (n, n). True = mask out.

    Returns:
        output: Attention output, shape (n, d_v)
        weights: Attention weights, shape (n, n)
    """
    d_k = Q.shape[-1]
    scores = Q @ K.T / np.sqrt(d_k)

    if mask is not None:
        scores = np.where(mask, -1e9, scores)

    weights = softmax(scores, axis=-1)
    output = weights @ V
    return output, weights

Out[16]:

Console

Full attention output shape: (5, 4)
Attention weights (first two rows):
[[0.1234 0.2426 0.2145 0.0651 0.3545]
 [0.2248 0.1815 0.158  0.2645 0.1712]]

This implementation handles the common case without masking. The optional mask argument will become important when we discuss causal attention in the next section. Notice that the function is entirely self-contained: given $Q$ , $K$ , and $V$ , it computes the output in four lines of code. The elegance of the implementation reflects the mathematical elegance of the formula itself.