Query, Key, Value › The QKV Mechanism: From Intuition to Formula

Now that we understand why we need separate representations for querying, being queried, and contributing content, let's build up the mathematical machinery that makes this work. We'll develop each component step by step, showing how the formulas directly address the challenge of learning flexible attention patterns.

The journey from intuition to formula follows a natural progression:

1. Project tokens into specialized QKV spaces
2. Measure compatibility between queries and keys using dot products
3. Scale the scores to prevent numerical instability
4. Aggregate values according to the attention weights

Each step solves a specific problem that arises from the previous one. By the end, you'll see how these pieces combine into a single elegant formula that powers modern transformers.

The core insight is simple: if we want tokens to play different roles, we should give them different representations for each role. We accomplish this through learned linear projections, separate transformations that map each token's embedding into QKV spaces.

Why three separate projections? Consider what happens when a pronoun like "it" needs to find its antecedent. The pronoun's embedding encodes "third-person singular neuter pronoun," but that's not what it should search for. It should search for "noun phrase that could be referred to." Meanwhile, a noun like "cat" shouldn't advertise "I'm a cat" but rather "I'm a noun phrase available for reference." And when "cat" contributes information, it should provide its semantic content (the concept of a cat), not its grammatical role.

These are three fundamentally different jobs:

• Querying: What information am I looking for?
• Being queried: What information do I have to offer?
• Contributing: What content should I transmit?

A single embedding can't optimally serve all three purposes. So we learn three different transformations, each specialized for its role.

Given an input embedding $\mathbf{x} \in \mathbb{R}^d$ for a single token, we compute:

$$\mathbf{q} = \mathbf{x} \mathbf{W}_Q, \quad \mathbf{k} = \mathbf{x} \mathbf{W}_K, \quad \mathbf{v} = \mathbf{x} \mathbf{W}_V$$

where:

• $\mathbf{x} \in \mathbb{R}^d$: the input embedding vector (a row vector with $d$ dimensions)
• $\mathbf{q} \in \mathbb{R}^{d_k}$: the resulting query vector, expressing what this token is looking for
• $\mathbf{k} \in \mathbb{R}^{d_k}$: the resulting key vector, advertising what this token contains
• $\mathbf{v} \in \mathbb{R}^{d_v}$: the resulting value vector, carrying the content to be aggregated
• $\mathbf{W}_Q \in \mathbb{R}^{d \times d_k}$: the query projection matrix (learned during training)
• $\mathbf{W}_K \in \mathbb{R}^{d \times d_k}$: the key projection matrix (learned during training)
• $\mathbf{W}_V \in \mathbb{R}^{d \times d_v}$: the value projection matrix (learned during training)
• $d$: the input embedding dimension
• $d_k$: the dimension of queries and keys (must match for dot product compatibility)
• $d_v$: the dimension of values (can differ from $d_k$)

Each projection is a matrix multiplication, a linear transformation where each output dimension is a weighted combination of input dimensions. These weights are learned during training. The model doesn't know in advance which embedding features matter for querying versus being queried versus contributing content. Instead, gradient descent discovers these patterns from data:

• $\mathbf{W}_Q$ learns which combinations of embedding features make effective queries (what patterns indicate "I need information about X")
• $\mathbf{W}_K$ learns which combinations make effective keys (what patterns indicate "I have information about X")
• $\mathbf{W}_V$ learns which content is worth transmitting when attention flows

This learning happens implicitly through the training objective. If attending from a verb to its subject improves next-word prediction, the projection matrices gradually adjust to make verb queries match subject keys.

Scaling to sequences. For a complete sequence of $n$ tokens, we stack all embeddings into a matrix $\mathbf{X} \in \mathbb{R}^{n \times d}$ (where each row is one token) and project all tokens simultaneously:

$$\mathbf{Q} = \mathbf{X} \mathbf{W}_Q, \quad \mathbf{K} = \mathbf{X} \mathbf{W}_K, \quad \mathbf{V} = \mathbf{X} \mathbf{W}_V$$

The resulting matrices have shapes $\mathbf{Q} \in \mathbb{R}^{n \times d_k}$, $\mathbf{K} \in \mathbb{R}^{n \times d_k}$, and $\mathbf{V} \in \mathbb{R}^{n \times d_v}$. Row $i$ of each matrix is the query, key, or value vector for token $i$.

Why might these projections learn different things? Consider an embedding that encodes both syntactic information (part of speech, grammatical role) and semantic information (meaning, topic). The query projection $\mathbf{W}_Q$ might learn to emphasize syntactic features, helping verbs find their subjects by grammatical role rather than semantic similarity. Meanwhile, the value projection $\mathbf{W}_V$ might preserve semantic content, so that when attention flows, it transfers meaning rather than grammatical markers.

The separation between keys and values is particularly powerful. A word's key determines which queries it matches (what attention it receives), while its value determines what information it contributes. This decoupling means a word can attract attention for one reason (syntactic role) while contributing entirely different information (semantic content).

With queries and keys in hand, we face the next challenge: how do we measure whether a query matches a key? We need a scoring function that takes two vectors and returns a single number indicating compatibility. Higher numbers should mean better matches.

Several options exist, including additive attention, multiplicative attention, and others, but the dot product has become the standard choice for its simplicity and efficiency. The geometric interpretation is intuitive:

• Vectors pointing in the same direction → large positive dot product (strong match)
• Orthogonal vectors → zero dot product (no relationship)
• Vectors pointing in opposite directions → negative dot product (poor match)

This gives us exactly the ranking behavior we want: the most compatible keys get the highest scores, and training can learn arbitrary matching patterns by adjusting the projection matrices.

For positions $i$ and $j$, the compatibility score is:

$$\text{score}_{ij} = \mathbf{q}_i \cdot \mathbf{k}_j = \sum_{m=1}^{d_k} q_{im} \cdot k_{jm}$$

where:

• $\text{score}_{ij}$: the raw attention score measuring how well position $i$'s query matches position $j$'s key
• $\mathbf{q}_i \in \mathbb{R}^{d_k}$: the query vector for position $i$
• $\mathbf{k}_j \in \mathbb{R}^{d_k}$: the key vector for position $j$
• $q_{im}$, $k_{jm}$: the $m$-th components of these vectors
• $d_k$: the dimension of query and key vectors (must match for the dot product to be defined)

The beauty of the dot product is that it's differentiable, fast to compute, and captures alignment in learned representation space. Because $\mathbf{q}_i$ and $\mathbf{k}_j$ come from learned projections, the model can discover arbitrary matching patterns during training.

To compute all $n^2$ scores at once (every query against every key), we use matrix multiplication:

$$\mathbf{S} = \mathbf{Q} \mathbf{K}^T$$

where:

• $\mathbf{S} \in \mathbb{R}^{n \times n}$: the score matrix containing all pairwise attention scores
• $\mathbf{Q} \in \mathbb{R}^{n \times d_k}$: the query matrix (row $i$ is the query for position $i$)
• $\mathbf{K}^T \in \mathbb{R}^{d_k \times n}$: the transposed key matrix (column $j$ is the key for position $j$)

Entry $S_{ij}$ is the dot product of row $i$ of $\mathbf{Q}$ with column $j$ of $\mathbf{K}^T$, which equals $\mathbf{q}_i \cdot \mathbf{k}_j$. This single matrix multiplication gives us all $n^2$ pairwise scores in one highly optimized operation, a key reason why attention scales well on modern hardware.

The power of learned projections. This is where QKV attention differs fundamentally from raw embedding similarity. In the previous chapter, tokens matched based on how similar their embeddings were, a fixed computation that couldn't adapt to context. Now they match based on how well their learned projections align.

Consider the implications: a verb's query can learn to match the key of a noun playing the subject role, even if "run" and "dog" have very different embeddings. The word "it" can learn a query that matches noun phrase keys, even though pronouns and nouns occupy different regions of embedding space. The projections transform the matching problem from "what words are similar?" to "what words are relevant?", and relevance is learned from data.

We have scores, but there's a subtle problem lurking in the mathematics. As the dimension $d_k$ grows, dot products tend to become larger in magnitude, not because the vectors are more aligned, but simply because we're summing more terms. This creates numerical instability that can derail training. To understand why and how we fix it, let's work through the statistics step by step.

Setting up the problem. Assume each component of $\mathbf{q}$ and $\mathbf{k}$ has zero mean and unit variance (a reasonable assumption for normalized embeddings). We want to understand how the variance of the dot product depends on dimension.

Step 1: Write the dot product as a sum. The dot product is:

$$\mathbf{q} \cdot \mathbf{k} = \sum_{m=1}^{d_k} q_m \cdot k_m$$

where:

• $q_m$: the $m$-th component of the query vector $\mathbf{q}$
• $k_m$: the $m$-th component of the key vector $\mathbf{k}$
• $d_k$: the dimension of both vectors

Step 2: Compute variance of each term. Each product $q_m \cdot k_m$ is the product of two independent random variables with zero mean and unit variance. For independent random variables $X$ and $Y$ with $\mathbb{E}[X] = \mathbb{E}[Y] = 0$ and $\text{Var}(X) = \text{Var}(Y) = 1$:

$$\text{Var}(X \cdot Y) = \mathbb{E}[X^2] \cdot \mathbb{E}[Y^2] = 1 \cdot 1 = 1$$

So each term $q_m \cdot k_m$ has variance 1.

Step 3: Sum the variances. For independent random variables, the variance of a sum equals the sum of variances:

$$\text{Var}\left(\sum_{m=1}^{d_k} q_m \cdot k_m\right) = \sum_{m=1}^{d_k} \text{Var}(q_m \cdot k_m) = \sum_{m=1}^{d_k} 1 = d_k$$

Conclusion. The dot product has variance $d_k$, so its standard deviation is $\sqrt{d_k}$. At $d_k = 64$ (common in practice), scores have standard deviation 8. At $d_k = 512$, it's about 22.6. This growth in magnitude isn't a feature; it's a bug that we need to fix.

The shapes confirm our understanding: input embeddings flow through projections to create QKV. The attention weight matrix is always $n \times n$ because we compute all pairwise interactions. The output has the same sequence length as the input but takes on the value dimension $d_v$. In practice, $d_v$ is often chosen to equal $d_k$ and the original embedding dimension, enabling residual connections.

We've now computed attention weights, a probability distribution over positions for each query. But weights alone don't produce useful outputs. The final step uses these weights to aggregate value vectors, producing new representations enriched with contextual information.

This is where the separation of keys and values proves its worth. Keys determined which positions receive attention (through query-key matching). Values determine what content flows. A word might attract attention because of its syntactic role (captured in its key) while contributing semantic information (carried in its value).

For position $i$, the output is a weighted sum of all value vectors:

$$\mathbf{o}_i = \sum_{j=1}^{n} \alpha_{ij} \mathbf{v}_j$$

where:

• $\mathbf{o}_i \in \mathbb{R}^{d_v}$: the output vector for position $i$, now enriched with contextual information
• $\alpha_{ij} \in [0, 1]$: the attention weight from position $i$ to position $j$, indicating how much position $i$ attends to position $j$
• $\mathbf{v}_j \in \mathbb{R}^{d_v}$: the value vector at position $j$, carrying the content that can be transferred
• $n$: the sequence length (total number of positions)
• $d_v$: the value dimension

The attention weights $\alpha_{ij}$ come from applying softmax to the scaled scores, so they satisfy $\sum_{j=1}^{n} \alpha_{ij} = 1$ for each position $i$. This constraint ensures the output is a proper weighted average.

Geometrically, this weighted average has an elegant interpretation. Position $i$'s output is a blend of all value vectors, with weights determined by query-key compatibility. Positions with keys that matched $i$'s query (high $\alpha_{ij}$) contribute more; positions with mismatched keys (low $\alpha_{ij}$) contribute less. Because the weights sum to 1, the output $\mathbf{o}_i$ lies within the convex hull of the value vectors, a point "inside" the space spanned by all possible values.

The information flow paradigm. Self-attention can be understood as a message-passing system. Each position broadcasts its value vector, and each position receives a custom blend of all broadcasts, with the blend weights determined by query-key compatibility. Unlike recurrent networks that pass information sequentially, or convolutional networks that only see local neighborhoods, attention allows any position to directly receive information from any other position in a single step.

This direct, content-addressable communication is what makes transformers so effective. A pronoun can directly access its antecedent, regardless of distance. A verb can simultaneously gather information from its subject plus object and modifiers. The network doesn't need to learn complex routing through intermediate positions; it learns which positions are relevant and attends to them directly.

Let's visualize the complete attention flow for a simple sentence: