Attention Intuition › How Attention Integrates with the Decoder

Understanding attention in isolation is one thing; understanding how it connects to the full decoding process is another. Let's trace through the entire decoder step in detail, seeing exactly how the context vector feeds into the generation of each output token.

At each decoding step $t$, the decoder maintains a hidden state $s_t$ and must produce two things: the next output token and the updated hidden state for the next step. The attention mechanism intervenes between these two operations, enriching the decoder's information before it makes the prediction.

The standard decoder with attention follows this sequence at each step $t$:

First, the decoder receives two inputs: the embedding of the previously generated token $y_{t-1}$ and the previous context vector $c_{t-1}$. These are concatenated and fed into the RNN cell together with the previous hidden state $s_{t-1}$:

$$s_t = \text{RNN}([y_{t-1}; c_{t-1}], s_{t-1})$$

Next, the new decoder state $s_t$ is used to compute attention over the encoder states, producing a fresh context vector $c_t$ tailored to the current generation step:

$$c_t = \text{Attention}(s_t, h_1, \ldots, h_T)$$

Finally, $s_t$ and $c_t$ are combined to predict the output token. A common approach is to concatenate them, apply a linear transformation to produce a pre-output vector $\tilde{s}_t$, and then apply softmax over the vocabulary:

$$\tilde{s}_t = \tanh(W_c [c_t; s_t])$$ $$P(y_t \mid y_{<t}, x) = \text{softmax}(W_v \tilde{s}_t)$$

where $W_c$ and $W_v$ are learned weight matrices. This is the architecture used in Luong attention, which we will study in detail in the Luong Attention chapter.

There is a noteworthy design choice in how the decoder state is computed. Some implementations, including Bahdanau's original formulation, compute attention using $s_{t-1}$ (the previous decoder state) rather than $s_t$ (the current one). Using $s_{t-1}$ for attention means the model selects its focus before integrating the previous output token, while using $s_t$ means the focus is informed by the current token's embedding. Both approaches work in practice, with Bahdanau using $s_{t-1}$ and Luong using $s_t$.

The full decoder loop creates an interesting feedback structure. The context vector $c_t$ influences what token is generated. That generated token becomes the input for the next step, which influences the new decoder state $s_{t+1}$, which in turn shapes the next attention distribution. Attention is not a one-shot computation; it participates in a dynamic, step-by-step dialogue between the decoder and the encoder states.

This feedback loop is why attention is so powerful for generation tasks. When the decoder generates accurate translations early in the output by focusing well, it creates a good hidden state for the next step. That good hidden state produces a better query for attention, which produces a better context vector, and so on. Conversely, when the decoder makes an error, that error can propagate forward through the hidden state, making subsequent attention queries less accurate. This is one reason why beam search (maintaining multiple hypotheses in parallel) is useful: it hedges against early errors by not committing to a single generation path.