Attention Intuition › A Step-by-Step Worked Example

To cement the mechanics, let's walk through attention computation for a complete sentence, step by step, with concrete numbers.

We will use the translation of "The cat sat" to French, simplified to focus on the attention computation. The encoder has processed the English sentence and produced three hidden states:

$$h_1 = [0.8, 0.2] \quad \text{("The")}$$ $$h_2 = [0.1, 0.9] \quad \text{("cat")}$$ $$h_3 = [0.6, 0.5] \quad \text{("sat")}$$

These are 2-dimensional vectors to keep the arithmetic readable. In practice they would be hundreds of dimensions.

Step t=1: Generating "Le" (The)

The decoder starts with an initial state $s_1 = [0.7, 0.3]$ (encoding "start of translation, producing an article").

Alignment scores using dot product:

$$e_{11} = s_1^\top h_1 = 0.7 \times 0.8 + 0.3 \times 0.2 = 0.56 + 0.06 = 0.62$$ $$e_{12} = s_1^\top h_2 = 0.7 \times 0.1 + 0.3 \times 0.9 = 0.07 + 0.27 = 0.34$$ $$e_{13} = s_1^\top h_3 = 0.7 \times 0.6 + 0.3 \times 0.5 = 0.42 + 0.15 = 0.57$$

Applying softmax to get attention weights (using $\exp$ of each score divided by the sum):

$$\exp(0.62) \approx 1.86, \quad \exp(0.34) \approx 1.40, \quad \exp(0.57) \approx 1.77$$ $$\text{sum} = 1.86 + 1.40 + 1.77 = 5.03$$ $$\alpha_{11} = 1.86 / 5.03 \approx 0.37, \quad \alpha_{12} = 1.40 / 5.03 \approx 0.28, \quad \alpha_{13} = 1.77 / 5.03 \approx 0.35$$

Context vector (weighted sum of encoder states):

$$c_1 = 0.37 \times [0.8, 0.2] + 0.28 \times [0.1, 0.9] + 0.35 \times [0.6, 0.5]$$ $$c_1 = [0.296, 0.074] + [0.028, 0.252] + [0.210, 0.175] = [0.534, 0.501]$$

The attention is fairly distributed here, which makes sense for generating an article: "Le" is mostly determined by the subject noun ("cat"), but generating it requires knowing the overall sentence structure.

Step t=2: Generating "chat" (cat)

After generating "Le," the decoder advances to state $s_2 = [0.2, 0.8]$ (now oriented toward producing the noun). The attention query shifts dramatically toward $h_2$:

$$e_{21} = 0.2 \times 0.8 + 0.8 \times 0.2 = 0.16 + 0.16 = 0.32$$ $$e_{22} = 0.2 \times 0.1 + 0.8 \times 0.9 = 0.02 + 0.72 = 0.74$$ $$e_{23} = 0.2 \times 0.6 + 0.8 \times 0.5 = 0.12 + 0.40 = 0.52$$

The score for position 2 ("cat") is now much higher. After softmax:

$$\alpha_{21} \approx 0.17, \quad \alpha_{22} \approx 0.53, \quad \alpha_{23} \approx 0.30$$

Now over half the attention weight falls on "cat." The context vector $c_2$ will be dominated by $h_2$. This provides the decoder with a strong signal about the noun to generate.

Step t=3: Generating the verb

For the verb phrase, the decoder state would shift again to attend primarily to $h_3$ ("sat"). This illustrates how attention tracks the linguistic content being generated at each step, automatically shifting focus as the translation proceeds through different parts of the source sentence.

This step-by-step trace reveals something important: the attention weights are not predetermined. They emerge from the interaction between the decoder's current state and the encoder states at each step. As the decoder evolves through the translation, its internal state changes, and those changes shift which encoder positions receive high attention. The mechanism is fully dynamic, computed fresh at every generation step.