Luong Attention › Summary

This chapter explored Luong attention, a family of attention mechanisms that simplified and extended Bahdanau's original formulation.

The key innovations include three score functions for computing alignment:

• Dot product: Parameter-free, efficient, requires matching dimensions
• General: Learned bilinear transformation, handles different dimensions
• Concat: Two-layer network with nonlinearity, most expressive

Luong attention also introduced the distinction between global and local attention. Global attention considers all encoder positions, while local attention focuses on a predicted window. Global attention dominates in practice due to its flexibility and the efficiency of modern hardware.

The architectural placement differs from Bahdanau: Luong computes attention after the RNN step using the current hidden state, then combines the context with the hidden state through a learned transformation. This "attention as output" approach is simpler and slightly more parallelizable.

Perhaps most importantly, Luong's work demonstrated that simple attention mechanisms, particularly dot product attention, can match or exceed more complex alternatives. This insight directly influenced the transformer architecture, which uses scaled dot-product attention as its core mechanism. Understanding Luong attention provides essential context for the self-attention mechanisms we'll explore in the next part of this book.