Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Query, Key, Value › QKV as Learned Transformations

Michael BrenndoerferUpdated June 4, 20254 min read

Language AI HandbookAll chapters

Part 13 of 67Self-Attention

Chapter 91Query, Key, Value › QKV as Learned Transformations

Reading section 5 of 11 · QKV as Learned Transformations · 4 minSection 5 of 11 · 4 min

Section 5 of 11 · 4 min read

Query, Key, Value › QKV as Learned TransformationsOne section · Sections in this chapter5 / 11 ↓

Sections in this chapter11

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Why do learned projections help? Consider what happens without them. If we use raw embeddings, a word can only attend to other words that happen to have similar embeddings. The word "it" might attend to "this" and "that" (similar pronouns) but struggle to attend to "cat" (semantically relevant but embedding-distant).

With learned projections, the model can discover arbitrary matching patterns. During training:

$\mathbf{W}_Q$ learns to transform embeddings into representations that express what each word is looking for
$\mathbf{W}_K$ learns to transform embeddings into representations that advertise what each word offers
$\mathbf{W}_V$ learns what content should flow when attention is paid

These projections operate independently, so a word's query doesn't need to resemble its key or value. A pronoun's query might encode "seeking a noun phrase antecedent" while its key might encode "available for coreference" and its value might encode its referent-neutral semantic content.

Let's visualize how the same embeddings project differently into QKV spaces:

In[17]:

Code

# Create a 3-word example with 2D projections for visualization
np.random.seed(42)
words_viz = ["cat", "sat", "mat"]

# Original embeddings (3D for variety, project to 2D)
embed_dim_viz = 4
d_proj = 2  # Project to 2D for visualization

X_viz = np.random.randn(3, embed_dim_viz)

# Different projection matrices for Q, K, V
W_q_viz = np.random.randn(embed_dim_viz, d_proj) * 0.5
W_k_viz = np.random.randn(embed_dim_viz, d_proj) * 0.5
W_v_viz = np.random.randn(embed_dim_viz, d_proj) * 0.5

# Project
Q_viz = X_viz @ W_q_viz
K_viz = X_viz @ W_k_viz
V_viz = X_viz @ W_v_viz

Out[18]:

Visualization

2D scatter plot showing three words projected into query space with different positions. — Query space: Where words look for information. The relative positions determine which queries match which keys.

2D scatter plot showing the same three words projected into key space with different arrangement. — Key space: How words advertise themselves. Words with keys near a query receive high attention from that query.

2D scatter plot showing the same three words projected into value space with yet another arrangement. — Value space: What content flows during attention. The geometric arrangement here determines what information gets aggregated.

The three projections arrange the same words differently. In query space, the positions reflect what each word is searching for. In key space, positions reflect how words present themselves to queries. In value space, positions determine what gets aggregated. A word might be close to another in query space (they look for similar things) but far in value space (they contribute different content).