Teacher Forcing › Combining Cross-Entropy and Reinforcement Learning

Michael BrenndoerferMay 21, 20252 min read
Language AI HandbookAll chapters
Part 12 of 67Sequence-to-Sequence
Chapter 84Teacher Forcing › Combining Cross-Entropy and Reinforcement Learning
Reading section 10 of 16 · Combining Cross-Entropy and Reinforcement Learning · 2 minSection 10 of 16 · 2 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 84 of 513

Section 10 of 16 · 2 min read

Teacher Forcing › Combining Cross-Entropy and Reinforcement LearningOne section · Sections in this chapter10 / 16 ↓
Sections in this chapter16
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

In modern practice, cross-entropy loss (teacher forcing) and REINFORCE-style rewards are rarely used in isolation. Instead, they are combined in a mixed training objective that exploits the strengths of both approaches. The standard formulation is:

Lmixed=λLCE+(1−λ)LRL\mathcal{L}_{\text{mixed}} = \lambda \mathcal{L}_{\text{CE}} + (1 - \lambda) \mathcal{L}_{\text{RL}}

where:

  • λ\lambda: the mixing coefficient, typically set between 0.9 and 0.99 to keep training stable
  • LCE\mathcal{L}_{\text{CE}}: the cross-entropy (teacher forcing) loss
  • LRL\mathcal{L}_{\text{RL}}: the policy gradient (REINFORCE) loss

The high value of λ\lambda reflects the practical wisdom that cross-entropy loss provides essential regularization. Without it, the RL component alone tends to overfit to the specific evaluation metric, sometimes in degenerate ways. Models trained only with BLEU score have been known to learn to repeat high-frequency phrases that boost BLEU without improving actual translation quality. The cross-entropy component keeps the model's predicted distribution over tokens close to the true distribution, which prevents degenerate reward-hacking behavior.

A common training recipe in industrial NMT systems is to first train to convergence with pure teacher forcing, then switch to the mixed objective for a limited number of additional epochs. The rationale is that teacher forcing is far more efficient during the initial phase, since it provides a clean and informative gradient signal at every step. Once the model has developed strong basic competence, the RL fine-tuning phase can safely optimize the task metric with reasonable stability.

This two-stage approach also addresses the cold-start problem for RL training. If you apply REINFORCE from the beginning, the model's initial random outputs receive very low rewards, and the gradient variance is enormous because the baseline (the greedy output) is also terrible. There is very little useful signal for the model to learn from. Starting from a strong teacher-forced checkpoint means the model already produces reasonable outputs, the rewards are in a useful range, and the advantage signal (difference between sampled and greedy rewards) is informative.