Teacher Forcing › Summary

Teacher forcing is a training technique that provides the correct previous token to the decoder at each step, rather than feeding back the model's own predictions. The key takeaways from this chapter are:

• Efficiency: Teacher forcing provides stable gradients and fast convergence because every training step conditions on correct context. It also enables fully parallel training in transformer architectures, which is essential for large-scale training.
• Exposure bias: The train-test mismatch arises because the model is never exposed to its own errors during training, but must handle them at inference time. This causes compounding errors during free-running decoding, especially for long sequences.
• Scheduled sampling: A curriculum learning approach that gradually transitions from teacher forcing to free-running by decaying the teacher forcing probability over training. The inverse sigmoid schedule is often the most effective because it preserves a stable initial learning phase.
• Professor forcing: An adversarial approach that aligns the model's hidden state distributions between teacher-forced and free-running modes, addressing exposure bias at the representation level rather than just the input level.
• REINFORCE and SCST: Policy gradient approaches that directly optimize task metrics. Self-critical sequence training uses the model's own greedy output as a variance-reducing baseline.
• Practical recipe: Most modern systems pretrain with teacher forcing for efficiency, then fine-tune with a mixed objective combining cross-entropy and reinforcement learning. This two-stage approach combines training efficiency with inference-time quality.
• Transformer robustness: Transformer-based models show less exposure bias than RNN-based models, likely due to their attention mechanisms and large training scales, but the fundamental train-test mismatch persists.

In the next chapter, we will look at beam search, the standard decoding algorithm used at inference time. Beam search partially compensates for exposure bias by maintaining multiple candidate hypotheses throughout decoding, but it introduces its own set of tradeoffs around beam width and length normalization.