Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Scaled Dot-Product Attention › Weighted Sum of Values

Michael BrenndoerferMay 25, 20253 min read

Language AI HandbookAll chapters

Part 13 of 67Self-Attention

Chapter 92Scaled Dot-Product Attention › Weighted Sum of Values

Reading section 5 of 15 · Weighted Sum of Values · 3 minSection 5 of 15 · 3 min

Section 5 of 15 · 3 min read

Scaled Dot-Product Attention › Weighted Sum of ValuesOne section · Sections in this chapter5 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The final step uses the attention weights to aggregate the value vectors. For each query position $i$ , we compute a weighted combination of all value vectors. This step produces the enriched representations: it takes information from all positions in the sequence and blends it together, with the blend controlled by the attention weights we just computed.

To understand why this design makes sense, consider the alternative: using the keys directly for the output instead of having a separate value projection. If we used keys as values, the same representation would serve two incompatible purposes: advertising relevance (key role) and contributing information (value role). Keeping them separate gives the model the freedom to have token $j$ 's key say "I am relevant to syntactic subject queries" while its value says "here is the rich semantic content I contribute when selected." This decoupling is what allows attention heads to specialize in different linguistic functions.

For each query position $i$ , the output is:

\text{output}_i = \sum_{j=1}^{n} A_{ij} \mathbf{v}_j

where:

$A_{ij}$ : the attention weight from position $i$ to position $j$ , giving the fraction of position $j$ 's value to include
$\mathbf{v}_j$ : the value vector for position $j$ , containing the information that position $j$ contributes when selected
$\text{output}_i$ : the attention output for position $i$ , a $d_v$ -dimensional context-enriched vector

In matrix form across all positions simultaneously:

\text{Attention}(Q, K, V) = AV = \text{softmax}\!\left(\frac{QK^T}{\sqrt{d_k}}\right) V

where:

$A \in \mathbb{R}^{n \times n}$ : the attention weight matrix, rows are probability distributions over positions
$V \in \mathbb{R}^{n \times d_v}$ : the value matrix, rows are value vectors for each position
$AV \in \mathbb{R}^{n \times d_v}$ : the output matrix, rows are attention outputs for each position

The output for each position is a soft lookup: a blended summary of information from all positions, weighted by relevance. Positions that received high attention weights contribute most; positions with low weights contribute little. The output has shape $(n, d_v)$ , matching the input shape (with $d_v$ instead of the original embedding dimension $d$ ). This means each token gets a new representation of the same dimension, enriched by context from the whole sequence.

Why is this better than a hard lookup (selecting the single most relevant position)? Because language rarely has crisp, discrete dependencies. Understanding "it" in "The cat sat on the mat and it purred" requires integrating information from multiple positions: "cat" is most relevant, but the verb tense and surrounding context also contribute. The soft weighted sum captures this gracefully. It is also differentiable everywhere, which means gradients flow smoothly through the output back to both the value vectors and the attention weights, allowing end-to-end training to optimize all components jointly.

In[13]:

Code

# Compute the attention output
output = A @ V  # (n, d_v)

Out[14]:

Console

Output shape: (5, 4)
Attention output matrix (each row is one token's enriched representation):
[[-0.0954 -0.66    0.6243 -0.1932]
 [-0.246  -0.6701  0.1977  0.2133]
 [-0.2605 -0.6365  0.1641  0.26  ]
 [-0.1539 -0.6517  0.4929 -0.0397]
 [-0.1038 -0.7851  0.2428 -0.0734]]

Each row of the output is a $d_v$ -dimensional vector: a context-aware representation of that token, blended from value vectors according to attention weights. Tokens that attended broadly have outputs that are smooth blends of many values; tokens that attended sharply to a few positions have outputs dominated by those positions' values. After this step, each token's representation contains information from the entire sequence, weighted by learned relevance, which is precisely what makes attention powerful for understanding language.