Bahdanau Attention › Limitations and Impact

Bahdanau attention changed how we think about sequence-to-sequence models. However, it has several limitations that motivated subsequent research.

The most significant limitation is the sequential nature of decoding. Because attention at step $i$ depends on the decoder state $s_{i-1}$, and $s_{i-1}$ depends on the attention at step $i-1$, we cannot parallelize across decoder timesteps. Each step must wait for the previous step to complete. This sequential bottleneck limits training throughput on modern parallel hardware like GPUs, where we'd prefer to process many positions simultaneously.

Another limitation is the additive score function's computational cost. Computing $\tanh(W_a s_{i-1} + U_a h_j)$ requires more operations than simpler alternatives like dot-product attention. While the difference is small for individual computations, it adds up over millions of training examples. Luong attention, which we'll cover in the next chapter, addresses this with more efficient score functions.

The attention mechanism also introduces additional hyperparameters (the alignment dimension $d_a$) and learnable parameters ($W_a$, $U_a$, $v_a$). While these provide flexibility, they also increase the risk of overfitting on small datasets and require careful tuning.

Despite these limitations, Bahdanau attention had a major impact on the field. It showed that attention mechanisms could improve performance on sequence-to-sequence tasks, reducing the BLEU score gap between neural and phrase-based machine translation systems. It also introduced the core ideas that would evolve into the self-attention mechanism at the heart of Transformers.

The key insights that carried forward include:

• Dynamic context: Computing a fresh context vector at each decoding step, rather than using a fixed representation
• Soft alignment: Learning to align input and output positions without explicit supervision
• Interpretability: Attention weights provide insight into model behavior
• Differentiable lookup: Treating attention as a soft, differentiable lookup table

These ideas laid the groundwork for the attention-based models that would reshape NLP over the following years.

A natural question is: how does attention learn to produce meaningful alignments? At initialization, the attention weights are random. Through training, the model gradually discovers which source positions are relevant for each target position.