Seq2Seq for MT (2014) › The Solution: Two Networks, One Conversation

The sequence-to-sequence breakthrough came from a deceptively simple idea: what if, instead of building a complex pipeline with a dozen components, you used just two neural networks that learned to work together? The first network.the encoder.would read the source sentence and figure out what it means. The second network.the decoder.would take that meaning and express it in the target language. That's it. No explicit word alignment. No phrase tables. No reordering rules. Just two networks learning to communicate through a bottleneck of numbers.

Think of it like a game of charades between two players who get better with practice. The encoder "acts out" the meaning of the source sentence by converting it into a fixed-size vector.essentially a list of numbers that represents everything important about the sentence. The decoder watches this performance and tries to recreate the meaning in a different language. During training, both networks learn together: the encoder learns how to compress meaning into that vector in ways the decoder can understand, and the decoder learns how to reconstruct meaning from it. They develop their own shared language of numbers.

The encoder's job is to read the source sentence and distill it down to its essence. It does this using a special type of neural network called an LSTM (Long Short-Term Memory) or GRU (Gated Recurrent Unit).architectures specifically designed to remember information as they process sequences.

Imagine the encoder reading "The cat sat on the mat" word by word. As it processes each word, it maintains an internal "state".a vector of numbers that represents everything it has understood so far:

1. Reads "The" → updates its internal state
2. Reads "cat" → updates state again, now knowing we're talking about a cat
3. Reads "sat" → state now encodes that a cat is doing something
4. Continues through "on", "the", "mat"

By the time the encoder finishes reading, its final state contains a compressed representation of the entire sentence. This vector needs to encode everything the decoder will need to reconstruct the meaning: what entities are involved (a cat, a mat), what action occurred (sitting), and what relationships exist between them (the cat is on the mat).

The magic of LSTMs is that they can remember information from early in the sentence even after reading many more words. Regular recurrent neural networks would forget what they saw at the beginning by the time they reached the end of a long sentence.they suffer from what's called the "vanishing gradient problem," where the training signal gets weaker and weaker as it propagates backward through time. But LSTMs use special gating mechanisms.think of them as valves that control information flow.that let them selectively remember or forget information, making them much better at handling long sequences.

Some researchers used bidirectional encoders that read the sentence both forward and backward, then combined both readings. This gave even richer representations because each word could be understood in the context of both what came before and what came after.like reading a mystery novel where you know the ending.

The decoder starts with the encoder's final state.that compressed representation of the source sentence.and uses it to generate the translation word by word. This process is called autoregressive generation because each new word depends on all the previous words, like writing a story where each sentence builds on what came before.

The translation of our example into French might proceed as follows:

1. Starts with the encoded representation and generates "Le" (The)
2. Uses the encoding + "Le" to generate "chat" (cat)
3. Uses the encoding + "Le chat" to generate "était" (was/sat)
4. Continues generating: "assis", "sur", "le", "tapis"

At each step, the decoder updates its own internal state, which tracks what it has generated so far and what it still needs to say. The decoder essentially maintains a running "memory" of the partial translation it's building.

The decoder doesn't just output a single word at each step.it outputs a probability distribution over the entire vocabulary. It might say "there's a 70% chance the next word should be 'Le', 15% chance it's 'Un', 10% chance it's 'La'..." During training, the system learns to assign high probability to the correct next word. During actual translation (called inference), the decoder typically picks the most likely word at each step, though more sophisticated decoding strategies like beam search can explore multiple possibilities simultaneously.

Seq2seq trains the two networks together. During training, you show the system pairs of sentences: "The cat sat on the mat" → "Le chat était assis sur le tapis". The system tries to translate the English sentence, and you measure how wrong it is using a mathematical loss function (specifically, cross-entropy loss). Then you adjust both the encoder and decoder simultaneously to reduce that error.

Mathematically, the system learns to maximize the probability $P(t | s)$ of generating the correct target sentence $t$ given source sentence $s$. This breaks down into a product of probabilities for each word: $P(t | s) = \prod_{i=1}^{|t|} P(t_i | s, t_{<i})$. In plain English: the probability of the complete translation equals the probability of the first target word, times the probability of the second word given the first, times the probability of the third word given the first two, and so on.

One training trick proved essential: teacher forcing. During training, when generating the second word of the translation, the decoder receives the actual correct first word as input.not its own (potentially wrong) prediction. This makes training much more efficient because the decoder always has the right context to learn from. It's like practicing piano with a teacher who corrects your mistakes immediately, rather than letting you practice errors. During inference (actual translation), the decoder must use its own predictions instead, which can sometimes lead to errors accumulating if it makes a mistake early on.

The beauty of end-to-end learning is that you're optimizing a single objective: make good translations. The encoder learns whatever representations are useful for that goal. The decoder learns whatever generation strategy works best. You don't need to specify how word alignment should work or what features matter.the networks figure that out themselves through the training process. It's learning by example rather than by rules.

Early attempts at seq2seq used simple recurrent neural networks, but they struggled with longer sentences. The problem was that gradients.the signals used to update the network during training.would either vanish to zero or explode to infinity as they propagated backward through many time steps. It was like trying to send a message through a long chain of people whispering to each other: by the time the message reached the end, it was either completely garbled or impossibly loud.

LSTMs solved this with their gating mechanisms. Think of gates as smart valves that control information flow. An LSTM has three types of gates:

• Forget gate: Decides what information to discard from the internal state
• Input gate: Decides what new information to add
• Output gate: Decides what to output based on the current state

These gates let the network learn to remember "cat" from the beginning of the sentence while processing "mat" at the end. Without this selective memory, translation quality degraded rapidly for sentences longer than about 10 words. LSTMs made it possible to handle sentences of 30, 40, even 50 words with reasonable quality.