Encoder-Decoder Framework › Decoder Architecture

The decoder generates the target sequence $y_1, y_2, \ldots, y_{T'}$ one token at a time, conditioned on the context vector. It is an autoregressive model: each output token depends on all previously generated tokens.

At each decoder timestep $t$, the decoder computes a hidden state based on the previous decoder hidden state, the previous output token, and the context vector. The initial decoder hidden state is set to the context vector $c$.

$$s_t = f_{\text{dec}}(y_{t-1}, s_{t-1}, c)$$

where:

• $s_t \in \mathbb{R}^d$ is the decoder hidden state at time $t$
• $y_{t-1}$ is the embedding of the previous target token (or the start-of-sequence token at $t = 1$)
• $s_{t-1}$ is the previous decoder hidden state (with $s_0 = c$)
• $c$ is the context vector from the encoder
• $f_{\text{dec}}$ is the decoder recurrent cell

A linear projection followed by a softmax then converts the decoder hidden state $s_t$ into a probability distribution over the target vocabulary:

$$P(y_t \mid y_{< t}, x) = \text{softmax}(W_o \cdot s_t + b_o)$$

where:

• $W_o \in \mathbb{R}^{V_{\text{tgt}} \times d}$ is the output projection matrix
• $b_o \in \mathbb{R}^{V_{\text{tgt}}}$ is the output bias
• $V_{\text{tgt}}$ is the target vocabulary size
• $P(y_t \mid y_{< t}, x)$ is the probability of each possible next token given all previous tokens and the source

The decoder generates tokens by sampling or taking the argmax of this distribution at each step. It continues until it produces the special end-of-sequence token <EOS>, after which generation stops.

The decoder generation process requires special tokens to signal the beginning and end of the sequence. A start-of-sequence token <SOS> (sometimes called <BOS>) is fed as the first decoder input at $t = 1$. When the decoder generates the end-of-sequence token <EOS>, generation terminates. During training, target sequences are wrapped with these tokens: the decoder input target sequence starts with <SOS> and the target labels end with <EOS>.

These special tokens play a specific semantic role. The <SOS> token is both a signal to begin generating and the initial "seed" that the decoder uses, in combination with the context vector, to choose the first target token. The decoder's representation of <SOS> in embedding space is learned during training, and it effectively encodes the concept of "beginning of sequence in this language with this context." Without a start token, the decoder has no way to make its first prediction, since its autoregressive design requires a previous token at every step.

The <EOS> token serves a similarly critical function. It gives the decoder a way to signal "I am done" rather than generating forever. At inference time, the generation loop checks for this token at every step and stops when it is produced. During training, the model is supervised to produce <EOS> at the correct position in the target sequence, so it learns to generate sequences of the right length.

In the most straightforward implementation, the context vector is used once: as the initial hidden state of the decoder. After the first decoder step, the context information is only accessible through the decoder's evolving hidden state, which must carry it forward across all decoding steps.

An alternative is to concatenate the context vector to the decoder's input at every step. Under this scheme, the decoder at each position sees both the embedding of the previous token and the full context vector:

$$s_t = f_{\text{dec}}([y_{t-1}; c], s_{t-1})$$

This gives the decoder a direct, unmediated connection to the context at each step, preventing the context information from degrading as the decoder hidden state evolves over many timesteps. Empirically, this variant tends to perform better on longer sequences, since the decoder does not have to rely solely on memory to retain the source summary.

A third approach concatenates the context vector to the output projection input, giving the vocabulary distribution direct access to the source encoding at every position. In practice, many implementations combine these ideas, concatenating context to both the input and the output projection.

At inference time, the simplest strategy is greedy decoding: at each step, the decoder selects the single highest-probability token and feeds it as input for the next step. This runs in $O(T')$ forward passes and is fast, but it is not guaranteed to find the globally most probable sequence.

The problem is that the greedy choice at step $t$ may close off more probable paths at steps $t+1, t+2, \ldots$. Consider a translation where the correct first word is "unfortunately," which might have probability 0.30 at step 1, but the second most probable word "sadly" has probability 0.25. Greedy decoding picks "unfortunately," but if "sadly" leads to a more probable continuation overall, greedy decoding would have made a suboptimal local decision.

Beam search addresses this by keeping the top $k$ candidate partial sequences at each step and expanding all of them before pruning back to the top $k$. This allows the decoder to explore multiple alternatives simultaneously, catching cases where a lower-probability early token leads to a higher-probability full sequence. The Beam Search chapter covers this algorithm in detail. For our implementation in this chapter, we use greedy decoding for simplicity and speed.