Luong Attention › Global vs Local Attention

Beyond score functions, Luong et al. introduced an important architectural distinction: global attention attends to all encoder states, while local attention focuses on a subset of positions around a predicted alignment point.

Global attention is conceptually straightforward. At each decoder timestep $t$, we compute attention weights over all encoder hidden states by applying softmax to the alignment scores. The attention weight for source position $s$ at decoder timestep $t$ is:

$$\alpha_{ts} = \frac{\exp(\text{score}(h_t, \bar{h}_s))}{\sum_{s'=1}^{S} \exp(\text{score}(h_t, \bar{h}_{s'}))}$$

where:

• $\alpha_{ts}$: the attention weight for source position $s$ when decoding at timestep $t$
• $\text{score}(h_t, \bar{h}_s)$: the alignment score computed using any of the three score functions
• $S$: the total number of encoder positions (source sequence length)
• $\exp(\cdot)$: the exponential function, which ensures all values are positive before normalization
• The denominator normalizes the weights so they sum to 1 across all source positions

The context vector is then computed as the weighted sum of all encoder states:

$$c_t = \sum_{s=1}^{S} \alpha_{ts} \bar{h}_s$$

where:

• $c_t \in \mathbb{R}^{d_{\bar{h}}}$: the context vector at decoder timestep $t$
• $\alpha_{ts}$: the attention weight for position $s$ (how much to focus on that position)
• $\bar{h}_s$: the encoder hidden state at position $s$

This weighted sum allows the decoder to "focus" on relevant source positions: positions with high attention weights contribute more to the context vector, while positions with low weights contribute less.

Global attention is what we typically mean when we say "attention" without qualification. It allows the model to attend to any position in the source sequence, which is essential for tasks like translation where word order can differ dramatically between languages.

The computational cost of global attention is $O(S)$ per decoder step, where $S$ is the source sequence length. For most NLP tasks with sequences of hundreds or thousands of tokens, this is manageable. However, for very long sequences (documents, books, genomic data), global attention becomes a bottleneck.

Local attention restricts the attention window to a subset of encoder positions centered around an alignment point $p_t$. This reduces computation and can also serve as an inductive bias when alignments are expected to be roughly monotonic.

Luong et al. propose two variants for determining the alignment point $p_t$:

Monotonic alignment (local-m): The alignment point is simply set to $p_t = t$, assuming the source and target sequences are roughly aligned. This works well for tasks like speech recognition where the output follows the input order.

Predictive alignment (local-p): The model learns to predict the alignment point using a small neural network. The predicted alignment point is:

$$p_t = S \cdot \sigma(v_p^\top \tanh(W_p h_t))$$

where:

• $p_t$: the predicted alignment point (center of the attention window) at decoder timestep $t$
• $S$: the source sequence length, used to scale the output to a valid position
• $\sigma(\cdot)$: the sigmoid function, which maps any real number to $[0, 1]$
• $W_p \in \mathbb{R}^{d_h \times d_h}$: a learned weight matrix that transforms the decoder hidden state
• $v_p \in \mathbb{R}^{d_h}$: a learned weight vector that projects to a scalar
• $h_t$: the current decoder hidden state

The sigmoid ensures the output is between 0 and 1, and multiplying by $S$ scales it to a valid source position. This allows the model to learn non-monotonic alignments while still restricting attention to a local window.

Within the window $[p_t - D, p_t + D]$, attention weights are computed normally but multiplied by a Gaussian centered at $p_t$:

$$\alpha_{ts} = \text{align}(h_t, \bar{h}_s) \cdot \exp\left(-\frac{(s - p_t)^2}{2\sigma^2}\right)$$

where:

• $\alpha_{ts}$: the final attention weight for position $s$ at decoder timestep $t$
• $\text{align}(h_t, \bar{h}_s)$: the base attention weight from softmax over the local window
• $s$: the source position being attended to
• $p_t$: the predicted center of the attention window
• $\sigma$: the standard deviation of the Gaussian (controls window sharpness), typically set to $D/2$
• The exponential term is a Gaussian that peaks at $s = p_t$ and decays for positions farther from the center

The Gaussian favors positions near the center of the window. This provides a soft boundary rather than a hard cutoff. Positions at the edge of the window receive lower weights even if their alignment scores are high.

The choice between global and local attention depends on your task:

• Global attention is the default choice for most NLP tasks. Translation, summarization, and question answering all benefit from the ability to attend to any position. The computational overhead is acceptable for typical sequence lengths.

• Local attention shines when you have prior knowledge about alignment structure. Speech recognition, where phonemes appear in order, is a natural fit. Document-level tasks with very long sequences can also benefit from the reduced computation.

In practice, global attention dominates because modern hardware handles the $O(S)$ computation efficiently, and the flexibility to attend anywhere is valuable. Local attention is more of a historical curiosity, though its ideas influenced later work on sparse attention patterns in transformers.