Attention Mechanism (2015) › The Solution

Bahdanau and his colleagues let the decoder access all encoder hidden states instead of only the final one. At each generation step, the model computed weights over those states and formed a step-specific context vector.

Different target words can depend on different source positions. The attention mechanism computed an alignment score between the decoder state and each encoder state, normalized the scores, and used them to form a weighted combination of encoder states.

The attention mechanism worked by computing alignment scores between the decoder's current hidden state and each encoder hidden state. For each position $i$ in the source sentence and the current decoder step $j$, the model computed an alignment score $e_{ij}$ that measured how well the source word at position $i$ aligned with the target word being generated at step $j$. These scores were computed using a small neural network, often called an alignment model, that took the decoder hidden state and encoder hidden state as inputs.

The alignment scores were then normalized using a softmax function to create attention weights. This normalization ensured that the weights summed to one and could be interpreted as a probability distribution over source positions. For decoder step $j$, the attention weight $\alpha_{ij}$ for source position $i$ was computed as:

$$\alpha_{ij} = \frac{\exp(e_{ij})}{\sum_{k=1}^{T} \exp(e_{kj})}$$

where $T$ is the length of the source sentence. These weights determined how much each encoder hidden state contributed to the context vector used for generating the current target word.

The context vector for decoder step $j$ was computed as a weighted sum of all encoder hidden states, where the weights came from the attention mechanism:

$$c_j = \sum_{i=1}^{T} \alpha_{ij} h_i$$

where $h_i$ represents the encoder hidden state at position $i$. This context vector contained information from all source positions, weighted by their relevance to the current decoding step, and was combined with the decoder's hidden state to generate the next target word.

Bahdanau's original paper proposed an additive attention mechanism, where the alignment score was computed using a feedforward network with a single hidden layer. The score $e_{ij}$ was calculated as:

$$e_{ij} = v^T \tanh(W_a s_{j-1} + U_a h_i)$$

where $s_{j-1}$ is the decoder hidden state at the previous step, $h_i$ is the encoder hidden state at position $i$, $W_a$ and $U_a$ are weight matrices, $v^T$ is a learned vector, and $\tanh$ is the activation function. This additive approach allowed the model to learn complex relationships between decoder and encoder states.

Later work, particularly by Minh-Thang Luong and colleagues, introduced a simpler dot-product attention that computed alignment scores directly from the inner product between transformed decoder and encoder states. This variant reduced computational complexity while maintaining similar performance:

$$e_{ij} = (W_q s_j)^T (W_k h_i)$$

where $W_q$ and $W_k$ are learned transformation matrices. The dot-product approach was simpler and faster to compute, making it attractive for practical applications while preserving the core attention mechanism's ability to learn dynamic alignments.

At each decoding step, the model computed weights over the encoder positions and formed a weighted context vector. It combined that context with the decoder hidden state to predict the next target word.

The weights could be plotted as an alignment matrix between source and target positions. In the running example, high values might connect "chat" with "cat," "s'est assis" with "sat," and "tapis" with "mat." These alignments were learned from the translation objective rather than direct alignment labels.