Query, Key, Value › A Worked Example with Real Words
Language AI HandbookAll chapters Query, Key, Value › A Worked Example with Real WordsOne section · Sections in this chapter7 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
To make these abstractions concrete, let's trace through QKV attention step by step with a meaningful sentence. We'll use deliberately small embeddings (4 dimensions) and projection matrices (projecting to 3 dimensions) so we can follow every number through the computation.
# Sentence: "The cat sat"
words_example = ["The", "cat", "sat"]
n = len(words_example)
# Create simple 4D embeddings (in practice, these come from an embedding layer)
# We'll make them somewhat interpretable:
# - Dimension 0: Determiner-ness
# - Dimension 1: Noun-ness
# - Dimension 2: Verb-ness
# - Dimension 3: Animacy
embeddings_example = np.array(
[
[1.0, 0.0, 0.0, 0.0], # "The" - determiner
[0.1, 1.0, 0.0, 0.8], # "cat" - animate noun
[0.0, 0.1, 1.0, 0.0], # "sat" - verb
]
)
# Small projection matrices (4D -> 3D)
np.random.seed(42)
d_k_ex = d_v_ex = 3
W_q_ex = np.random.randn(4, d_k_ex) * 0.5
W_k_ex = np.random.randn(4, d_k_ex) * 0.5
W_v_ex = np.random.randn(4, d_v_ex) * 0.5Original Embeddings: Word Embed ---------------------------------------- The [1. 0. 0. 0.] cat [0.1 1. 0. 0.8] sat [0. 0.1 1. 0. ]
We've designed these embeddings to be interpretable: each dimension corresponds to a linguistic feature. "The" is a pure determiner (1.0 only in the determiner dimension). "cat" is primarily a noun with high animacy. "sat" is a pure verb. In real systems, embeddings would be dense vectors where meaning is distributed across dimensions, but our hand-crafted features help us trace what the projections do.
Step 1: Computing QKV Projections. The projection matrices transform these 4D embeddings into 3D QKV vectors:
# Step 1: Project to Q, K, V
Q_ex = embeddings_example @ W_q_ex
K_ex = embeddings_example @ W_k_ex
V_ex = embeddings_example @ W_v_exStep 1: QKV Projections ================================================== Query vectors (what each word is looking for): The Q = [ 0.248 -0.069 0.324] cat Q = [ 1.003 -0.309 -0.271] sat Q = [ 0.866 0.372 -0.246] Key vectors (how each word presents itself): The K = [ 0.121 -0.957 -0.862] cat K = [-0.359 -0.575 -0.499] sat K = [-0.482 -0.757 0.749] Value vectors (what each word contributes): The V = [-0.272 0.055 -0.575] cat V = [-0.262 0.034 -0.692] sat V = [-0.282 0.896 -0.021]
Observe that each word now has three different representations. The same embedding for "cat" becomes different vectors in QKV spaces. This is the core of the QKV framework: specialized representations for specialized roles.
With random projection matrices (as we have here), the vectors don't encode anything linguistically meaningful. But the structure is in place: if we trained these projections on language data, "cat"'s query might learn to seek modifiers and related nouns, its key might learn to signal "available as subject/object," and its value might carry semantic features worth transmitting.
Step 2: Computing Scaled Attention Scores. Now we compute how well each query matches each key:
# Step 2: Compute scaled attention scores
raw_scores = Q_ex @ K_ex.T
scaled_scores = raw_scores / np.sqrt(d_k_ex)
Step 2: Attention Scores
==================================================
Raw scores (Q @ K^T):
The cat sat
The -0.183 -0.211 0.175
cat 0.651 -0.047 -0.452
sat -0.039 -0.402 -0.883
Scaled scores (÷ √3 = ÷ 1.732):
The cat sat
The -0.106 -0.122 0.101
cat 0.376 -0.027 -0.261
sat -0.022 -0.232 -0.510The raw scores are dot products between query and key vectors. Each entry measures how well one word's query matches another word's key. Scaling by compresses the score range, preventing the softmax saturation we discussed earlier.
Notice the variety in scores: some query-key pairs produce positive scores (the vectors point in similar directions), while others produce negative scores (opposing directions). This variety is what allows attention to be selective, with some positions receiving high attention while others receive little.
Step 3: Converting Scores to Attention Weights. Softmax maps these real-valued scores into a probability distribution:
# Step 3: Softmax to get attention weights
def softmax_rows(x):
exp_x = np.exp(x - x.max(axis=1, keepdims=True))
return exp_x / exp_x.sum(axis=1, keepdims=True)
attention_ex = softmax_rows(scaled_scores)
Step 3: Attention Weights (after softmax)
==================================================
The cat sat
The 0.311 0.306 0.383 (sum: 1.000)
cat 0.455 0.304 0.241 (sum: 1.000)
sat 0.412 0.334 0.253 (sum: 1.000)Each row now sums to exactly 1.0, making these valid probability distributions. The exponential in softmax amplifies differences: a score of 0.5 doesn't just beat 0.3 by a little. It gets substantially more weight after exponentiation. This creates soft but focused attention patterns.
Reading the table: "The" attends to all three words with weights shown in its row. "cat" distributes its attention differently. "sat" has its own pattern. These weights will determine how value vectors are blended to produce each word's output.
Step 4: Aggregating Values into Outputs. Finally, we use the attention weights to compute weighted sums of value vectors:
# Step 4: Weighted sum of values
output_ex = attention_ex @ V_ex
Step 4: Output (weighted sum of values)
==================================================
Each output is a blend of all value vectors:
The output = [-0.273 0.371 -0.399]
≈ 0.31×V_The + 0.31×V_cat + 0.38×V_sat
cat output = [-0.272 0.251 -0.477]
≈ 0.46×V_The + 0.30×V_cat + 0.24×V_sat
sat output = [-0.271 0.261 -0.474]
≈ 0.41×V_The + 0.33×V_cat + 0.25×V_satEach word's output is now a blend of all value vectors, weighted by the attention distribution. The equations show which contributions dominate, displaying only weights above 0.1 to focus on the meaningful contributions.
This is the essence of self-attention: each position receives a custom mixture of information from the entire sequence. The original embedding for "sat" knew nothing about what subject performed the action. After attention, "sat"'s representation incorporates information from "cat" (and "The"), creating a contextualized representation that encodes "sat with cat as subject in a determined noun phrase."
In trained models, these patterns become linguistically meaningful: verbs attend to their arguments, modifiers attend to what they modify, and pronouns attend to their antecedents, all learned automatically from data without explicit linguistic supervision.