Encoder-Decoder Framework › Worked Example: A Sequence Transformation

Let us trace through a small example to make the mechanics concrete. Suppose we want to learn a mapping from phrases to their word-reversed versions: "the cat sat" maps to "sat cat the".

Encoding phase:

The source tokens are ["<SOS>", "the", "cat", "sat", "<EOS>"]. The encoder embedding layer maps each token index to a 64-dimensional vector. The LSTM encoder processes these embeddings left to right, updating its hidden state and cell state at each step.

After reading <SOS>, the hidden state $h_1$ contains information about the start of a sequence, which is fairly generic. After reading "the", $h_2$ begins to reflect that the sequence starts with a common function word. After "cat", $h_3$ knows about "the cat," a likely subject phrase. After "sat", $h_4$ reflects a complete verb phrase "the cat sat." After <EOS>, $h_5$ contains the full context of a short sentence describing a sitting cat.

The encoder produces hidden states $h_1, h_2, h_3, h_4, h_5$. After processing the final <EOS> token, we take $c = (h_5, s_5)$ as the context vector pair.

Decoding phase:

The decoder starts with hidden state $s_0 = c$ and first input $y_0 = \text{embedding}(\text{<SOS>})$. At each step:

• $t = 1$: The decoder computes $s_1 = \text{LSTM}(y_0, s_0)$, projects to logits, and predicts "sat" as the highest-probability token.
• $t = 2$: Feed the embedding of "sat", compute $s_2$, and predict "cat".
• $t = 3$: Feed the embedding of "cat", predict "the".
• $t = 4$: Feed the embedding of "the", predict <EOS>.

The final output is "sat cat the", which is the correct word-reversed version.

During training with teacher forcing, the decoder input at $t = 2$ would be the correct token "cat" regardless of what the decoder predicted at $t = 1$.

What the context vector must encode:

For this word-reversal task, the context vector must capture the complete ordered sequence of content words in the source, so the decoder can emit them in reverse order. This is a particularly demanding test of the context vector because it requires retaining precise positional information about every word. In real translation, the context vector does not need to store verbatim source content; it needs to store the meaning, which is a more compressed representation. Even so, for real translation of moderately long sentences, the context vector quickly becomes the limiting factor.