Attention Intuition › Comparing Attention Formulations

We've established that attention requires a scoring function to measure relevance between decoder and encoder states. Different choices lead to different attention mechanisms, each with distinct trade-offs.

The fundamental question each scoring function answers is the same: "Given what I'm trying to generate (the decoder state) and what information is available (an encoder state), how compatible are they?" The answer is always a single number, the alignment score. But how we compute that number varies significantly.

The most direct approach treats compatibility as geometric alignment. Two vectors pointing in similar directions have high compatibility; orthogonal vectors have zero compatibility. The dot product captures exactly this intuition:

$$e_{tj} = s_t^\top h_j$$

where:

• $s_t$: the decoder state at step $t$, a vector of dimension $d$
• $h_j$: the encoder state at position $j$, also dimension $d$
• $s_t^\top h_j$: the inner product, computed as $\sum_{i=1}^{d} s_{t,i} \cdot h_{j,i}$

Geometrically, the dot product equals $\|s_t\| \cdot \|h_j\| \cdot \cos(\theta)$, where $\theta$ is the angle between the vectors. When both vectors point in the same direction ($\theta = 0$), the score is maximized. When they are perpendicular ($\theta = 90^\circ$), the score is zero.

This simplicity is both a strength and a limitation. Dot product attention requires no learnable parameters, making it computationally efficient. However, it requires the decoder and encoder to have the same hidden dimension, and it assumes compatibility can be measured purely through geometric alignment in the shared representation space. If the encoder and decoder use different architectures or different hidden sizes, dot product scoring cannot be applied without first projecting one into the other's space.

What if simple geometric alignment isn't expressive enough? Perhaps compatibility depends on complex, nonlinear relationships between the decoder and encoder states. Additive attention addresses this by introducing a small neural network:

$$e_{tj} = v^\top \tanh(W_s s_t + W_h h_j)$$

where:

• $W_s$: a learnable weight matrix of dimension $d_a \times d_s$ that projects the decoder state
• $W_h$: a learnable weight matrix of dimension $d_a \times d_h$ that projects the encoder state
• $v$: a learnable weight vector of dimension $d_a$ that produces the final scalar score
• $\tanh$: the hyperbolic tangent, introducing nonlinearity
• $d_a$: the attention hidden dimension, a hyperparameter

The formula works as follows: $W_s s_t$ projects the decoder state into a new space of dimension $d_a$. Similarly, $W_h h_j$ projects the encoder state into the same space. Adding these projections combines information from both states. The $\tanh$ activation introduces nonlinearity. Finally, $v^\top$ projects the combined representation to a scalar score.

This approach has two key advantages. First, the learnable parameters allow the model to discover what "compatibility" means for the specific task. The network can learn to ignore certain dimensions of the encoder or decoder state, to focus on dimensions that are systematically predictive of relevance, and to capture interaction effects between specific encoder and decoder features. Second, since we project both states into a common space of dimension $d_a$, the encoder and decoder can have different dimensions. We'll implement this fully in the Bahdanau Attention chapter.

The cost is additional parameters: $d_a \times d_s$ (for $W_s$) plus $d_a \times d_h$ (for $W_h$) plus $d_a$ (for $v$). For a typical setup with $d_a = 128$, $d_s = 256$, and $d_h = 256$, this adds about 66,000 parameters. Compared to the millions of parameters in the LSTM layers, this is modest. But the scoring function must also be evaluated at every encoder position for every decoder step, so its computational cost scales with sequence length.

The transformer architecture revived dot product attention but added a scaling refinement. The problem with vanilla dot products becomes apparent in high dimensions: scores can grow very large, causing softmax to produce extremely peaked distributions.

To understand why, consider what happens when we compute $s_t^\top h_j$ for vectors with $d_k$ dimensions. If each component is independently drawn from a zero-mean, unit-variance distribution, the variance of the dot product is approximately $d_k$. For $d_k = 512$, the standard deviation of scores is about 22. Scores this large cause softmax to assign nearly all probability mass to a single position, producing near-zero gradients for all other positions and slowing learning. The model gets "stuck" because the peaked softmax means nearly all gradient signal goes to the single highest-scoring position; the other positions receive almost no gradient and cannot improve.

Scaled dot product attention fixes this by normalizing:

$$e_{tj} = \frac{s_t^\top h_j}{\sqrt{d_k}}$$

where:

• $d_k$: the dimension of the key vectors
• $\sqrt{d_k}$: the scaling factor that brings variance back to approximately 1 regardless of dimension

The visualization confirms the scaling problem. As dimension increases, the distribution of dot product scores spreads dramatically. At $d=512$, scores routinely exceed $\pm 40$, which would cause softmax to produce extremely peaked distributions. Scaling by $\sqrt{d_k}$ normalizes this spread back to a manageable range, keeping the standard deviation near 1 regardless of embedding dimension. This simple fix makes scaled dot product attention stable across a wide range of hidden dimensions and is one reason why transformers could be scaled to very large sizes without requiring careful tuning of the attention mechanism.

Each approach represents a different trade-off:

Dot product attention is fastest but requires matching encoder and decoder dimensions. Additive attention is most flexible and introduces learned parameters, at the cost of more computation per position. Scaled dot product combines the efficiency of dot products with numerical stability for high dimensions.

In modern practice, scaled dot product attention dominates. Its efficiency allows transformers to run multiple attention heads in parallel, each learning different aspects of relevance. The lack of learnable parameters in the score function is offset by the projection matrices that create queries, keys, and values. When you apply a linear projection to the encoder states before computing attention, you are effectively learning the alignment metric, just in a different, more modular way. We'll explore this architecture in the Self-Attention chapters.