Luong Attention › Luong vs Bahdanau: A Complete Comparison
Language AI HandbookAll chapters Luong Attention › Luong vs Bahdanau: A Complete ComparisonOne section · Sections in this chapter5 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's consolidate the differences between these two influential attention mechanisms:
| Aspect | Bahdanau | Luong |
|---|---|---|
| Score function | Additive (concat with tanh) | Dot, general, or concat |
| Decoder state used | Previous () | Current () |
| Attention placement | Before RNN (input) | After RNN (output) |
| Context integration | Concatenated with input | Combined via learned layer |
| Encoder architecture | Bidirectional RNN | Unidirectional (stacked) |
| Parameters | More (additive scoring) | Fewer (dot product option) |
Both mechanisms compute attention weights via softmax and produce a context vector as a weighted sum of encoder states. The differences lie in the details of scoring, timing, and integration.
In practice, the choice between Bahdanau and Luong attention often matters less than other architectural decisions like model size, number of layers, and training procedure. Modern transformer architectures have largely superseded both, but understanding these mechanisms provides essential intuition for how attention works.