Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Query, Key, Value › Dimensions and Shapes

Michael BrenndoerferUpdated June 4, 20252 min read

Language AI HandbookAll chapters

Part 13 of 67Self-Attention

Chapter 91Query, Key, Value › Dimensions and Shapes

Reading section 4 of 11 · Dimensions and Shapes · 2 minSection 4 of 11 · 2 min

Section 4 of 11 · 2 min read

Query, Key, Value › Dimensions and ShapesOne section · Sections in this chapter4 / 11 ↓

Sections in this chapter11

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Understanding the tensor shapes in self-attention is important for implementation and debugging. Let's summarize the key dimensions:

Tensor shapes in self-attention. Query and key dimensions must match for dot product compatibility.
Tensor	Shape	Description
Input $\mathbf{X}$	$(n, d)$	$n$ tokens, each with $d$ -dimensional embedding
$\mathbf{W}_Q$	$(d, d_k)$	Query projection, maps $d \to d_k$
$\mathbf{W}_K$	$(d, d_k)$	Key projection, maps $d \to d_k$
$\mathbf{W}_V$	$(d, d_v)$	Value projection, maps $d \to d_v$
Queries $\mathbf{Q}$	$(n, d_k)$	Query vectors for all tokens
Keys $\mathbf{K}$	$(n, d_k)$	Key vectors for all tokens
Values $\mathbf{V}$	$(n, d_v)$	Value vectors for all tokens
Scores $\mathbf{Q}\mathbf{K}^T$	$(n, n)$	All pairwise attention scores
Weights $\alpha$	$(n, n)$	Softmax-normalized attention
Output	$(n, d_v)$	Context-enriched representations

A few important constraints:

Query and key dimensions must match ( $d_k$ ) because we compute their dot product
Value dimension can differ ( $d_v$ ) since values are aggregated, not compared
Output dimension equals value dimension ( $d_v$ ) because the output is a weighted sum of values

In practice, most implementations set $d_k = d_v = d / h$ , where $d$ is the embedding dimension and $h$ is the number of attention heads. This choice ensures that the total computation across all heads remains comparable to a single-head attention with full dimension. We'll explore multi-head attention in a later chapter.