Attention Intuition › The Mathematics of Attention

Now that we have the intuition, let's translate it into precise mathematics. The goal is to build a mechanism that answers one question at each decoding step: "Which parts of the input should I focus on right now?"

Imagine you're the decoder, trying to generate the next word in a translation. You have your current state $s_t$, which encodes what you've generated so far and what you're trying to produce next. Meanwhile, the encoder has processed the entire input sentence and produced a sequence of hidden states $h_1, h_2, \ldots, h_T$, one for each of the $T$ input words. Each encoder state $h_j$ captures the meaning of word $j$ in context.

The challenge is clear: you need to selectively combine information from all these encoder states, giving more weight to positions that are relevant to your current generation task. Attention solves this in three steps.

The first question attention must answer is: "How relevant is each input position to what I'm currently generating?" We need a way to compare the decoder's current state with each encoder state and produce a relevance score.

This comparison happens through a scoring function:

$$e_{tj} = \text{score}(s_t, h_j)$$

where:

• $e_{tj}$: the alignment score, a single number indicating how relevant encoder position $j$ is when generating at decoder position $t$
• $s_t$: the decoder's hidden state at step $t$, encoding the generation context
• $h_j$: the encoder's hidden state at position $j$, encoding information about input word $j$
• $\text{score}(\cdot, \cdot)$: any function that takes two vectors and returns a scalar measuring their compatibility

Think of this as the decoder asking each encoder position: "How useful are you for what I'm trying to do right now?" A high score means "very useful," while a low or negative score means "not relevant."

What should this scoring function look like? The simplest choice is the dot product: $e_{tj} = s_t^\top h_j$, which measures how aligned the two vectors are in the embedding space. Vectors pointing in similar directions yield high scores; orthogonal vectors yield zero. We'll explore more sophisticated scoring functions in the section on attention formulations below, and cover them in full detail in the Bahdanau Attention and Luong Attention chapters.

The decoder state $s_t$ deserves a closer look. In an RNN decoder, $s_t$ is the hidden state at generation step $t$, after the decoder has already generated tokens $y_1, y_2, \ldots, y_{t-1}$. It carries information about both what has been generated and what the decoder is about to produce. This duality is why $s_t$ makes a good query: it simultaneously knows what context has already been established (so it doesn't ask the encoder to repeat what was already encoded in earlier context vectors) and what needs to be generated next (so it can seek out the relevant encoder information for the upcoming token).

Raw alignment scores present a problem: they can be any real number, positive or negative, large or small. We need to convert them into something interpretable and usable, specifically a probability distribution over input positions.

The softmax function accomplishes this transformation. Given the alignment scores $e_{t1}, e_{t2}, \ldots, e_{tT}$ for all $T$ input positions, the attention weight for position $j$ is:

$$\alpha_{tj} = \frac{\exp(e_{tj})}{\sum_{k=1}^{T} \exp(e_{tk})}$$

where:

• $\alpha_{tj}$: the attention weight for position $j$, guaranteed to be between 0 and 1
• $\exp(e_{tj})$: the exponential function applied to the alignment score, converting any real number to a positive value
• $\sum_{k=1}^{T} \exp(e_{tk})$: the sum of all exponentiated scores, serving as a normalizing constant

Why softmax? The exponential function maps any real number to a positive value. This ensures all weights are non-negative. Dividing by the sum of all exponentials guarantees that $\sum_{j=1}^{T} \alpha_{tj} = 1$. The result is a valid probability distribution over input positions.

Softmax also has a useful amplification property. If one score is much larger than the others, its corresponding weight will dominate. For example, if $e_{t1} = 5$ and all other scores are near 0, then $\alpha_{t1}$ will be close to 1 while the other weights approach 0. This allows the model to focus sharply on a single position when appropriate, or spread attention across multiple positions when relevance is more evenly distributed.

The competition enforced by softmax is worth dwelling on. Because all weights must sum to 1, increasing attention to one position automatically decreases attention to others. This makes attention a zero-sum competition for relevance: every additional unit of focus on position $j$ is taken from other positions. As a result, training the model to focus correctly on position $j$ requires rewarding high scores for $j$ while learning to suppress all positions that are less relevant. This competitive dynamic helps attention produce sharp, interpretable patterns rather than diffuse or arbitrary distributions.

The visualization above demonstrates this amplification effect. With uniform scores, softmax produces uniform weights (0.20 each). Small score differences get amplified into clearer preferences. Large differences produce nearly one-hot attention, where almost all weight concentrates on the highest-scoring position.

With attention weights in hand, we can finally answer our original question: "What information from the input should I use?" The answer is a weighted combination of all encoder states:

$$c_t = \sum_{j=1}^{T} \alpha_{tj} h_j$$

where:

• $c_t$: the context vector at decoder step $t$, a single vector summarizing the relevant input information
• $\alpha_{tj}$: the attention weight for encoder position $j$ at decoder step $t$
• $h_j$: the encoder hidden state at position $j$ (the "value" being retrieved)

This weighted sum is the heart of attention. Each encoder state contributes to the context vector in proportion to its attention weight. If $\alpha_{t3} = 0.7$ and all other weights are small, then $c_t$ will be dominated by $h_3$, the information at position 3. If weights are more evenly spread, the context vector blends information from multiple positions.

The context vector has the same dimension as the encoder states, making it easy to integrate with the rest of the model. The decoder uses $c_t$ alongside its own state $s_t$ to predict the next output token, often by concatenating them and passing through a feed-forward layer. The standard approach concatenates them as $[c_t; s_t]$ and passes the combined vector through a learned transformation followed by a softmax over the vocabulary.

One subtle but important property of this formula is that the context vector is always the same dimension as the encoder states, regardless of $T$, the number of input positions. Whether we are weighting 5 encoder states or 500, the result is always a single vector of dimension $d_h$. This is what gives attention its elegant scalability: adding more input tokens does not change the interface between attention and the rest of the model.

Let's trace through a concrete example. Suppose we're translating "The cat sat" and currently generating the French word "chat" (cat). The decoder state $s_t$ encodes that we've generated "Le" and are now producing the noun.

1. Alignment scores: The scoring function compares $s_t$ with each encoder state. It finds high compatibility with $h_2$ (the representation of "cat") and lower compatibility with $h_1$ ("The") and $h_3$ ("sat").

2. Attention weights: Softmax converts these scores into probabilities. Perhaps $\alpha_{t1} = 0.05$, $\alpha_{t2} = 0.85$, $\alpha_{t3} = 0.10$.

3. Context vector: The weighted sum $c_t = 0.05 \cdot h_1 + 0.85 \cdot h_2 + 0.10 \cdot h_3$ produces a context vector dominated by the representation of "cat."

The decoder combines $c_t$ with $s_t$ and predicts "chat" as the next word. At the next step, when generating "noir" (black), the attention mechanism shifts focus to the encoder state for "black." This dynamic, step-by-step focus is what makes attention effective. Rather than relying on a single compressed representation of the entire input, the model can look back at the original encoder states and select the information most relevant to each generation step.

Notice that this also handles an important linguistic phenomenon: reordering. In French, adjectives typically follow the noun, so "black cat" becomes "chat noir." Without attention, the decoder would have to somehow use a single compressed vector to generate words in a different order than the input. With attention, the decoder simply attends to "cat" when generating "chat" and to "black" when generating "noir," regardless of the fact that these words appeared in reverse order in the source. Attention handles word reordering naturally because it is agnostic to position, caring only about relevance.