Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Self-Attention Concept › Building Intuition with a Worked Example

Michael BrenndoerferUpdated February 6, 202610 min read

Language AI HandbookAll chapters

Part 13 of 67Self-Attention

Chapter 90Self-Attention Concept › Building Intuition with a Worked Example

Reading section 8 of 13 · Building Intuition with a Worked Example · 10 minSection 8 of 13 · 10 min

Section 8 of 13 · 10 min read

Self-Attention Concept › Building Intuition with a Worked ExampleOne section · Sections in this chapter8 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The formulas become clearer when we trace through a concrete example step by step. Let's process a three-word sequence, "I love coffee," using tiny 2-dimensional embeddings. With only 2 dimensions, we can easily verify each calculation by hand and visualize what's happening geometrically.

We'll assign each word an embedding that points in a different direction:

In[13]:

Code

# Three-word sequence with 2D embeddings
words = ["I", "love", "coffee"]
embeddings_2d = np.array(
    [
        [1.0, 0.0],  # "I" - points along first axis
        [0.5, 0.5],  # "love" - between axes (45 degrees)
        [0.0, 1.0],  # "coffee" - points along second axis
    ]
)

These embeddings form a simple geometric arrangement: "I" points east, "coffee" points north, and "love" points northeast, lying exactly between them. This setup will help us understand how the dot product captures similarity.

The geometric interpretation is instructive. The dot product $\mathbf{x}_i \cdot \mathbf{x}_j$ can be written as $|\mathbf{x}_i| |\mathbf{x}_j| \cos\theta$ , where $\theta$ is the angle between the two vectors. Vectors pointing in the same direction ( $\theta = 0$ ) have dot product equal to the product of their magnitudes. Perpendicular vectors ( $\theta = 90°$ ) have dot product zero. Vectors pointing in opposite directions have negative dot products. Our toy embeddings are set up so that "I" and "coffee" are perpendicular, making their dot product zero, while "love" makes a 45-degree angle with each, giving it identical dot products with both.

Out[14]:

Visualization

2D plot with three vectors as arrows from origin: I pointing right, coffee pointing up, love pointing diagonally. — 2D embedding space showing the three word vectors. ''I'' points east along the x-axis, ''coffee'' points north along the y-axis, and ''love'' lies at 45 degrees between them. The geometric arrangement determines the dot product similarities: perpendicular vectors like ''I'' and ''coffee'' have zero dot product, while ''love'' has equal moderate similarity to both.

The geometric arrangement makes the dot product intuition clear: vectors pointing in similar directions have high dot products, while perpendicular vectors have zero dot product.

Step 1: Computing Similarity ScoresLink Copied

First, we compute the dot product between every pair of words. Remember, the dot product $\mathbf{x}_i \cdot \mathbf{x}_j$ measures how much two vectors point in the same direction.

In[15]:

Code

# Compute similarity scores (dot products)
scores_2d = embeddings_2d @ embeddings_2d.T

Out[16]:

Console

Similarity scores (dot products):
                 I     love   coffee
I          1.00    0.50    0.00
love       0.50    0.50    0.50
coffee     0.00    0.50    1.00

Let's interpret this matrix:

Diagonal entries (1.00, 0.50, 1.00): These are self-similarities. "I" and "coffee" have unit-length embeddings, so their self-dot-products are 1.0. "love" has length $\sqrt{0.5^2 + 0.5^2} = \sqrt{0.5} \approx 0.71$ , giving a self-dot-product of 0.5.
"I" vs "coffee" (0.00): These vectors are perpendicular (orthogonal), so their dot product is zero. In embedding space, this means they share no directional similarity.
"love" vs others (0.50): "love" points between "I" and "coffee," giving it moderate similarity to both. The dot product of 0.5 reflects this intermediate relationship.

The asymmetry between "love" and itself (0.50) versus "I" and "coffee" and themselves (1.00) comes from vector length. The dot product is not normalized, so longer vectors have higher self-similarity. This is one reason why the scaled dot product version of self-attention (which we will study in the next chapter) sometimes normalizes embedding lengths. In this toy example, the unnormalized dot product is sufficient to see the main intuition.

Step 2: Applying SoftmaxLink Copied

Raw dot products aren't suitable weights for averaging because they don't sum to 1. The softmax function maps each row into a probability distribution:

In[17]:

Code

# Apply softmax to get attention weights
def softmax_rows(x):
    exp_x = np.exp(x - x.max(axis=1, keepdims=True))
    return exp_x / exp_x.sum(axis=1, keepdims=True)


attention_2d = softmax_rows(scores_2d)

Out[18]:

Console

Attention weights (after softmax):
                 I     love   coffee
I         0.506   0.307   0.186
love      0.333   0.333   0.333
coffee    0.186   0.307   0.506

Now each row sums to 1.0, and we can interpret the values as "attention percentages":

"I" attends most to itself (about 58%), moderately to "love" (31%), and least to "coffee" (11%). This makes sense: "I" had the highest dot product with itself, moderate with "love," and zero with "coffee."
"love" distributes attention more evenly across all three words (about 33% each). Its embedding lies between the others, giving it similar dot products with everyone.
"coffee" mirrors "I": it attends mostly to itself, moderately to "love," and barely to "I."

The softmax has transformed our raw similarities into a meaningful attention distribution. Higher similarity scores become higher attention weights, but even zero-similarity pairs get some weight (the exponential of 0 is 1, not 0). This is an important property: softmax never produces exactly zero weights. Every token always attends at least a little to every other token, which prevents the model from completely ignoring any part of the input. The practical implication is that the information flow in self-attention is always fully dense, not sparse.

Notice also the symmetry in this particular example: the attention weight from "I" to "coffee" (0.11) equals the weight from "coffee" to "I" (0.11). This symmetry arises because our scores matrix is symmetric: $s_{ij} = \mathbf{x}_i \cdot \mathbf{x}_j = \mathbf{x}_j \cdot \mathbf{x}_i = s_{ji}$ . In the full query-key-value formulation of self-attention (covered in the next chapter), separate learned projections break this symmetry, allowing "I" attending to "coffee" to have a different weight than "coffee" attending to "I." This asymmetry is linguistically meaningful: a pronoun attending to its antecedent is not the same relationship as the antecedent attending to the pronoun.

Step 3: Computing Contextual OutputsLink Copied

Finally, we use these attention weights to compute a weighted average of all embeddings for each position:

In[19]:

Code

# Compute output representations
output_2d = attention_2d @ embeddings_2d

Out[20]:

Console

Output representations:
I: [0.660, 0.340]
love: [0.500, 0.500]
coffee: [0.340, 0.660]

Original vs Output comparison:
I: [1.0, 0.0] -> [0.660, 0.340]
love: [0.5, 0.5] -> [0.500, 0.500]
coffee: [0.0, 1.0] -> [0.340, 0.660]

The transformation is subtle but meaningful. Each word's representation has shifted toward the other words, with the amount of shift determined by the attention weights:

"I" started at [1.0, 0.0] and moved to approximately [0.73, 0.27]. It picked up some "northward" component from "love" and "coffee."
"love" started at [0.5, 0.5] and stayed close to its original position. Because it attended fairly evenly to all words, and it was already in the "middle," the weighted average didn't change it much.
"coffee" started at [0.0, 1.0] and moved to approximately [0.27, 0.73]. It picked up some "eastward" component from "I" and "love."

This is the essence of self-attention: each word's output is no longer just its own embedding, but a blend of all words' embeddings, weighted by relevance. The word "I" now "knows about" the context of "love" and "coffee." This contextual blending is what enables transformers to build rich, context-dependent representations.

In a real model with high-dimensional embeddings and learned projections, this blending is much more expressive. The model can choose to gather very specific types of information from context: syntactic information from one head, semantic information from another, positional information from a third. But the fundamental mechanism is the same weighted average you see in this toy example.

Out[21]:

Visualization

2D plot showing original and transformed embeddings with arrows indicating the shift direction for each word. — How self-attention transforms embeddings in the 'I love coffee' example. Original positions (hollow circles) shift toward the weighted average of all embeddings, producing contextual outputs (filled circles). 'I' and 'coffee' move substantially toward each other via the intermediate 'love' vector, while 'love' barely shifts because it was already centrally positioned.

The visualization captures the essence of self-attention geometrically. Each word's representation moves toward the center of mass of all words, weighted by attention. "I" and "coffee" shift significantly toward each other (via "love"), while "love" barely moves because it was already centrally positioned. After this transformation, all three words carry information about the full context.

What the Example RevealsLink Copied

This worked example, despite its simplicity, illustrates several important properties of self-attention that generalize to real models.

First, the outputs are always convex combinations of the inputs. Because attention weights are non-negative and sum to 1, the output $\mathbf{y}_i$ is a weighted average: it lies in the convex hull of all input embeddings. This means self-attention cannot produce representations that are "outside" the range of the inputs; it can only blend and interpolate. This is both a strength (it is a stable, well-behaved operation) and a limitation (it cannot extrapolate beyond what is present in the input). The feed-forward layers that follow self-attention in a transformer block are responsible for introducing the non-linearity and capacity to produce outputs that go beyond simple blending.

Second, the transformation is context-dependent. The attention weights, and therefore the output representations, change with every different input sequence. Run the same three-word sequence through a random permutation (say, "coffee I love") and the attention weights will be different, and the output representations will be different. This context-dependence is what makes self-attention fundamentally more powerful than any fixed transformation.

Third, the mechanism is symmetric in the simplified version here, but asymmetric in the full QKV formulation. When queries and keys are computed by separate learned projections (rather than using the raw embeddings for both), the relationship "how much does position $i$ attend to position $j$ " can differ from "how much does position $j$ attend to position $i$ ." This asymmetry is linguistically important and is one reason why the full QKV formulation is more expressive.