Luong Attention › Attention Score Functions

The simplest approach to measuring relevance is to ask: how similar are the decoder and encoder representations? If the encoder has learned to represent "cat" in a particular way, and the decoder has learned to look for that same pattern when generating "chat", then similar representations should yield high scores.

The dot product provides exactly this similarity measure. Given a decoder hidden state $h_t$ and an encoder hidden state $\bar{h}_s$, the dot product attention score is:

$$\text{score}(h_t, \bar{h}_s) = h_t^\top \bar{h}_s$$

where:

• $h_t \in \mathbb{R}^{d}$: the decoder hidden state at timestep $t$, encoding what the decoder is "looking for"
• $\bar{h}_s \in \mathbb{R}^{d}$: the encoder hidden state at source position $s$, encoding what that position "contains"
• $h_t^\top \bar{h}_s$: the inner product of the two vectors, yielding a scalar score
• $d$: the hidden dimension, which must be equal for both encoder and decoder

Why does the dot product capture similarity? Consider what happens geometrically. When two vectors point in the same direction, their dot product is large and positive, indicating high compatibility. When they point in opposite directions, the dot product is large and negative. Orthogonal vectors, sharing no common direction, yield zero.

This geometric interpretation has a powerful implication: the encoder and decoder can learn complementary representations. The encoder learns to place semantically similar words in similar directions in the hidden space. The decoder learns to "query" for specific semantic content by producing hidden states that point toward the relevant encoder representations.

The elegance of dot product attention lies in its simplicity. No learned parameters are needed for the score function itself. All the learning happens in the encoder and decoder, which discover representations where raw similarity is a good proxy for alignment. This makes dot product attention computationally efficient and easy to implement.

However, this simplicity comes with a constraint: the decoder and encoder hidden states must have the same dimensionality. If $h_t \in \mathbb{R}^{d_h}$ and $\bar{h}_s \in \mathbb{R}^{d_h}$, the dot product is well-defined. If they have different dimensions, you cannot compute a dot product directly, which motivates our next score function.

What if the encoder and decoder have different hidden dimensions? Or what if raw similarity isn't the right measure, and we want the model to learn a more specific notion of relevance?

The general score function addresses both concerns by introducing a learnable weight matrix $W_a$ that transforms the encoder states before computing similarity:

$$\text{score}(h_t, \bar{h}_s) = h_t^\top W_a \bar{h}_s$$

where:

• $h_t \in \mathbb{R}^{d_h}$: the decoder hidden state at timestep $t$
• $\bar{h}_s \in \mathbb{R}^{d_{\bar{h}}}$: the encoder hidden state at source position $s$
• $W_a \in \mathbb{R}^{d_h \times d_{\bar{h}}}$: a learned weight matrix that projects encoder states into the decoder's space
• $h_t^\top W_a \bar{h}_s$: a scalar score computed by first applying $W_a$ to $\bar{h}_s$, then taking the dot product with $h_t$

To understand what $W_a$ does, trace through the computation step by step. First, $W_a \bar{h}_s$ converts the encoder state into a $d_h$-dimensional vector. Then $h_t^\top$ takes the dot product with this transformed vector. The matrix $W_a$ effectively learns which aspects of the encoder representation are most relevant for each dimension of the decoder state.

This formulation is called bilinear attention because the score is a bilinear function of the two input vectors: linear in $h_t$ when $\bar{h}_s$ is fixed, and linear in $\bar{h}_s$ when $h_t$ is fixed. The weight matrix $W_a$ serves two purposes:

• Dimension matching: When encoder and decoder have different hidden dimensions, $W_a$ bridges the gap by projecting encoder states into the decoder's space
• Learned similarity: Rather than relying on raw vector similarity, the matrix learns what aspects of encoder states are most relevant for alignment, potentially discovering non-obvious relationships

General attention is more expressive than dot product attention but requires additional parameters. For encoder and decoder dimensions of 512, $W_a$ adds $512 \times 512 = 262{,}144$ parameters. This is modest compared to the total model size but can matter for smaller models or when memory is constrained.

The most expressive score function takes a fundamentally different approach. Rather than computing a form of similarity between two vectors, it asks: given both the decoder state and an encoder state, what score should a neural network assign?

The concat score function concatenates the two states and passes them through a small neural network:

$$\text{score}(h_t, \bar{h}_s) = v_a^\top \tanh(W_a [h_t; \bar{h}_s])$$

Let's unpack this formula piece by piece:

1. Concatenation $[h_t; \bar{h}_s]$: We stack the decoder and encoder states into a single vector of dimension $d_h + d_{\bar{h}}$. This gives the network access to all information from both states.

2. Linear projection $W_a [h_t; \bar{h}_s]$: The weight matrix $W_a \in \mathbb{R}^{n \times (d_h + d_{\bar{h}})}$ projects the concatenated vector to an intermediate space of dimension $n$. This allows the network to learn arbitrary linear combinations of the input features.

3. Nonlinearity $\tanh(\cdot)$: The hyperbolic tangent activation introduces nonlinearity, enabling the network to learn complex, non-linear relationships between decoder and encoder states.

4. Scalar projection $v_a^\top$: Finally, the weight vector $v_a \in \mathbb{R}^n$ projects the intermediate representation to a single scalar score.

where:

• $h_t \in \mathbb{R}^{d_h}$: the decoder hidden state at timestep $t$
• $\bar{h}_s \in \mathbb{R}^{d_{\bar{h}}}$: the encoder hidden state at source position $s$
• $[h_t; \bar{h}_s] \in \mathbb{R}^{d_h + d_{\bar{h}}}$: the concatenated vector of both states
• $W_a \in \mathbb{R}^{n \times (d_h + d_{\bar{h}})}$: a learned projection matrix
• $v_a \in \mathbb{R}^n$: a learned weight vector that collapses the intermediate representation to a scalar
• $n$: the intermediate dimension, a hyperparameter typically set equal to the hidden dimension

This formulation is similar to Bahdanau attention, with one key difference: Bahdanau uses the previous decoder state $h_{t-1}$, while Luong's concat uses the current decoder state $h_t$. This seemingly small change has clear implications for the overall architecture, which we'll explore in the section on attention placement.

The three score functions differ significantly in their computational requirements. For a sequence of length $S$ with hidden dimension $d$:

As shown in Table score-functions, dot product attention is most efficient but requires matching dimensions. General attention adds a single matrix, while concat attention requires two sets of parameters ($W_a$ and $v_a$).