Scaled Dot-Product Attention › A Complete Worked Example
Language AI HandbookAll chapters Scaled Dot-Product Attention › A Complete Worked ExampleOne section · Sections in this chapter11 / 15 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Theory becomes concrete when you trace through a small example by hand. Let's work through a minimal but complete attention computation with a 4-token sequence: "the", "cat", "sat", "there". We'll use small dimensions () so every intermediate value fits on screen and we can interpret each step.
The goal of this example is to make the abstract operations tangible. When we say "query attends to key ," what does that look like numerically? When we say "the output is a weighted blend of values," what blend weights are we computing? Working through these questions with concrete numbers builds the intuition needed to understand attention in larger models.
Think of this worked example as running the attention formula in slow motion: we pause at each step to print the intermediate matrices, check their shapes, and interpret what the numbers mean before moving to the next step.
# Fixed random seed for reproducibility
np.random.seed(7)
# 4-token sequence, d=6 embedding dim, d_k=d_v=3
tokens = ["the", "cat", "sat", "there"]
n_ex = 4
d_ex = 6
dk_ex = 3
# Simulated learned embeddings for each token
X_ex = np.random.randn(n_ex, d_ex)
# Simulated learned projection weights
W_Q_ex = np.random.randn(d_ex, dk_ex) * 0.5
W_K_ex = np.random.randn(d_ex, dk_ex) * 0.5
W_V_ex = np.random.randn(d_ex, dk_ex) * 0.5
# Project to Q, K, V
Q_ex = X_ex @ W_Q_ex
K_ex = X_ex @ W_K_ex
V_ex = X_ex @ W_V_ex
# Step 1: Raw scores
S_ex = Q_ex @ K_ex.T
# Step 2: Scale
S_scaled_ex = S_ex / np.sqrt(dk_ex)
# Step 3: Softmax
A_ex = softmax(S_scaled_ex, axis=1)
# Step 4: Weighted sum of values
out_ex = A_ex @ V_ex=== Step 1: Raw scores QK^T ===
the cat sat there
the 3.977 3.688 2.635 3.222
cat 4.704 5.336 2.664 4.826
sat -2.616 -1.913 -2.144 -0.687
there 3.107 2.737 2.534 0.223
=== Step 2: Scaled scores (divide by sqrt(3)=1.732) ===
the cat sat there
the 2.296 2.129 1.521 1.861
cat 2.716 3.081 1.538 2.786
sat -1.510 -1.105 -1.238 -0.396
there 1.794 1.580 1.463 0.129
=== Step 3: Attention weights (softmax) ===
the cat sat there
the 0.3385 0.2865 0.1560 0.2190
cat 0.2617 0.3769 0.0806 0.2808
sat 0.1458 0.2187 0.1915 0.4441
there 0.3683 0.2974 0.2646 0.0697
=== Step 4: Output (A @ V) shape: (4, 3) ===
the: [-1.5223 -0.9709 0.1333]
cat: [-1.8853 -0.8206 0.1676]
sat: [-1.7339 -0.792 0.4547]
there: [-1.1056 -1.034 0.0768]Walk through what happened at each step. The raw score matrix in Step 1 contains unscaled query-key dot products: each entry is the raw similarity between one token's query and one token's key, before any normalization. After scaling in Step 2, the values shrink by a factor of , bringing them into the range where softmax operates without saturation. The attention weight matrix in Step 3 shows which positions each token attends to most: each row is a proper probability distribution, and you can see which keys received the most attention weight from each query. Finally, Step 4 shows the output for each token, which is a 3-dimensional vector representing that token's context-enriched representation after blending in information from across the sequence.
One observation worth making: even though the token sequence has semantic structure, these attention patterns are driven by random projection matrices rather than learned ones. In a trained model, the patterns would be far more meaningful: "the" might strongly attend to the noun it modifies ("cat"), "sat" might attend to the subject ("cat") and object ("there"), and so on. The random weights here demonstrate the mechanics; training is what makes the weights encode linguistic structure.