Bahdanau Attention › Summary

Bahdanau attention solves the bottleneck problem in encoder-decoder models by allowing the decoder to dynamically focus on relevant parts of the input at each generation step. The mechanism works through a learned alignment model that scores how well each encoder position matches the current decoder state.

The key components are:

• Additive score function: $e_{ij} = v_a^\top \tanh(W_a s_{i-1} + U_a h_j)$ computes alignment scores by projecting the decoder state $s_{i-1}$ and encoder states $h_j$ into a shared alignment space, combining them additively, and reducing to a scalar
• Softmax normalization: $\alpha_{ij} = \frac{\exp(e_{ij})}{\sum_k \exp(e_{ik})}$ converts raw scores to attention weights that form a probability distribution over encoder positions
• Context vector: $c_i = \sum_j \alpha_{ij} h_j$ computes a weighted sum of encoder states, where the weights $\alpha_{ij}$ determine how much each position contributes to the context for generating output $i$

The attention weights are interpretable, showing which input positions the model considers relevant for each output position. This interpretability, combined with strong empirical performance, made Bahdanau attention a foundational technique in neural machine translation and sequence-to-sequence modeling more broadly.

In the next chapter, we'll explore Luong attention, which offers alternative score functions with different computational trade-offs.