Teacher Forcing › The Training Challenge: Autoregressive Decoding

To understand why teacher forcing was invented, consider what happens without it. A seq2seq decoder operates autoregressively: each output token depends on all previously generated tokens. During inference, this is unavoidable, since you do not know what the correct output is. You simply feed each generated token back as input for the next step.

During training, you have the correct target sequence. So you face a choice that is more consequential than it first appears.

Free-running (fully autoregressive) training feeds the decoder's own previous predictions as input at each step. If the decoder generates a wrong token early in the sequence, that error propagates forward. The next step receives a wrong input, which makes the next output more likely to be wrong, and so on. The model must learn to recover from its own mistakes without any external guidance.

Teacher forcing replaces the decoder's previous prediction with the correct target token. At step $t$, the decoder receives the ground-truth token $y_{t-1}$ rather than its own prediction $\hat{y}_{t-1}$. The model never has to deal with the cascading consequences of its own errors during training.

Why does cascading matter so much? Consider training a model to translate a twenty-word sentence. If the model makes an error at position 3 and that error shifts the decoder into an incorrect hidden state, then positions 4 through 20 all receive a corrupted context. The gradients flowing back through those positions all carry signal about how to respond to corrupted states rather than about how to correctly translate the language. The model spends most of its learning budget on a task it will never face in a fully-supervised setting: error recovery. This makes training both slow and unstable.

Teacher forcing eliminates this problem entirely. Every gradient update teaches the model about the correct conditional distribution conditioned on correct context. Learning is fast and the training signal is clean.

The mathematical framing clarifies what each approach optimizes. In standard language modeling, the loss at each step is the cross-entropy between the predicted distribution and the true next token. With teacher forcing, the loss at step $t$ is:

$$\mathcal{L}_t = -\log P(\hat{y}_t = y_t \mid y_1, y_2, \ldots, y_{t-1}, \mathbf{c})$$

where:

• $y_t$: the true target token at step $t$
• $y_1, \ldots, y_{t-1}$: the ground-truth tokens at all previous steps
• $\mathbf{c}$: the encoder context vector (or set of context vectors with attention)
• $P(\hat{y}_t = y_t \mid \ldots)$: the decoder's predicted probability for the correct token

The total training loss sums over all time steps:

$$\mathcal{L} = -\sum_{t=1}^{T} \log P(\hat{y}_t = y_t \mid y_1, \ldots, y_{t-1}, \mathbf{c})$$

This is clean and efficient because every conditional probability is conditioned on correct context, making gradients stable and informative throughout the sequence. The gradients from step 20 flow back cleanly to earlier parts of the network without being corrupted by the effects of accumulated errors.

Without teacher forcing, the loss would instead be conditioned on the model's own predictions, which change as training progresses. This creates a non-stationary training signal that is much harder to optimize. The model is essentially trying to hit a moving target: the distribution it needs to learn from depends on its own current quality, so as the model improves, the training distribution shifts. This instability is one reason free-running training from scratch is rarely used for seq2seq models.