Luong Attention › Attention Placement: Input vs Output

A subtle but important difference between Bahdanau and Luong attention is where the attention mechanism fits into the decoder architecture. This choice affects both the information flow and the computational graph.

In Bahdanau attention, the context vector is computed using the previous decoder hidden state and then concatenated with the input embedding to form the input to the current decoder step. The computation proceeds as:

$$\begin{aligned} c_t &= \text{Attention}(h_{t-1}, \bar{H}) \\ \tilde{x}_t &= [x_t; c_t] \\ h_t &= \text{RNN}(\tilde{x}_t, h_{t-1}) \end{aligned}$$

where:

• $h_{t-1}$: the decoder hidden state from the previous timestep
• $\bar{H} = [\bar{h}_1, \bar{h}_2, \ldots, \bar{h}_S]$: the matrix of all encoder hidden states
• $c_t$: the context vector computed by attending over encoder states using $h_{t-1}$
• $x_t$: the input embedding at timestep $t$ (typically the previous output token)
• $\tilde{x}_t$: the augmented input formed by concatenating $x_t$ and $c_t$
• $h_t$: the new decoder hidden state

The context vector influences the RNN computation directly. This means the decoder can use information about which source positions are relevant when updating its hidden state.

In Luong attention, the decoder RNN runs first, producing the current hidden state. Then attention is computed using this new state, and the results are combined. The computation proceeds as:

$$\begin{aligned} h_t &= \text{RNN}(x_t, h_{t-1}) \\ c_t &= \text{Attention}(h_t, \bar{H}) \\ \tilde{h}_t &= \tanh(W_c [c_t; h_t]) \end{aligned}$$

where:

• $x_t$: the input embedding at timestep $t$
• $h_{t-1}$: the decoder hidden state from the previous timestep
• $h_t$: the new decoder hidden state after the RNN step
• $\bar{H} = [\bar{h}_1, \bar{h}_2, \ldots, \bar{h}_S]$: the matrix of all encoder hidden states
• $c_t$: the context vector computed by attending over encoder states using the current $h_t$
• $[c_t; h_t]$: the concatenation of context and hidden state
• $W_c \in \mathbb{R}^{d_h \times (d_{\bar{h}} + d_h)}$: a learned weight matrix that combines context and hidden state
• $\tilde{h}_t$: the "attentional hidden state" used for prediction

The final output $\tilde{h}_t$ combines the context vector with the decoder state through a learned transformation. This attentional state is then used for prediction, typically by projecting it to vocabulary size and applying softmax.

The placement choice has several practical implications:

Information flow: Bahdanau attention allows the context to influence the RNN state update, potentially enabling richer interactions. Luong attention keeps the RNN computation separate, which can be easier to reason about and debug.

Parallelization: Luong attention is slightly more amenable to parallelization during training because the RNN step doesn't depend on the attention computation. However, this advantage is minimal compared to the sequential nature of RNNs themselves.

Empirical performance: Luong et al. found their approach performed comparably or slightly better than Bahdanau attention on machine translation benchmarks. The simpler architecture and faster score functions (especially dot product) made it an attractive choice.