Teacher Forcing › REINFORCE for Sequence-to-Sequence Models

A fundamentally different approach to bridging the train-test gap is to directly optimize the evaluation metric using reinforcement learning. The REINFORCE algorithm (Williams, 1992) applied to seq2seq models treats the decoder as a policy that takes actions (token choices) to maximize a reward signal.

The motivation for this approach is straightforward but important. Cross-entropy loss with teacher forcing measures how well the model predicts each token given correct context. But the thing you care about in machine translation or summarization is not per-token prediction accuracy; it is the quality of the complete output sequence as measured by metrics like BLEU (for translation) or ROUGE (for summarization). These metrics are not decomposable step by step; they measure properties of the entire generated sequence. Teacher forcing cannot directly optimize them.

The key idea in REINFORCE-based training is to define the reward $R(\hat{y}, y)$ as the task-specific evaluation score between the generated sequence $\hat{y}$ and the ground-truth sequence $y$. For machine translation, this could be BLEU score. For summarization, this could be ROUGE. For dialogue, it could be a human preference score.

The policy gradient update for the model parameters $\theta$ is:

$$\nabla_\theta \mathcal{L}_{\text{RL}} = -\mathbb{E}_{\hat{y} \sim p_\theta}\left[R(\hat{y}, y) \cdot \nabla_\theta \log p_\theta(\hat{y} \mid \mathbf{x})\right]$$

where:

• $\hat{y} \sim p_\theta$: a complete output sequence sampled from the model
• $R(\hat{y}, y)$: the reward signal comparing generated output to ground truth
• $\nabla_\theta \log p_\theta(\hat{y} \mid \mathbf{x})$: the gradient of the log probability of the sampled sequence

The intuition is: sample a complete output sequence, evaluate how good it was using the task metric (the reward), then update parameters to make good sequences more likely and bad sequences less likely. This directly aligns the training objective with the evaluation objective, which is the fundamental appeal of the approach.

However, REINFORCE alone is unstable in practice due to high variance in the gradient estimates. Consider what happens when you sample a sequence and measure its BLEU score. The BLEU score of a single sentence is noisy: a sequence might score low due to random word choice differences from the reference, not fundamental quality issues. The gradient is weighted by this noisy reward, so the updates are highly variable. With pure REINFORCE, training often fails to converge or converges to poor solutions.

Rennie et al. (2017) introduced the self-critical training sequence (SCST) approach, which uses the model's own greedy decoding output as a baseline to reduce variance. Rather than using the raw reward, the update uses the advantage: how much better or worse was the sampled sequence compared to the greedy sequence?

$$\nabla_\theta \mathcal{L}_{\text{SCST}} = -\mathbb{E}_{\hat{y} \sim p_\theta}\left[(R(\hat{y}, y) - R(\bar{y}, y)) \cdot \nabla_\theta \log p_\theta(\hat{y} \mid \mathbf{x})\right]$$

where $\bar{y}$ is the greedy decoding output. This baseline reduces variance because the gradient is non-zero only when the sampled sequence is better or worse than the greedy output. If both sequences receive the same reward, no update is applied. This means the model is only updated when sampling discovers something the greedy decoding missed (positive advantage) or when it stumbles into something worse (negative advantage). The updates carry meaningful signal rather than noise.

The self-critical baseline has an elegant property: it is computed using the same model being trained, so it adapts automatically as the model improves. As the greedy output quality increases over training, the baseline rises, maintaining a useful performance gap that keeps the learning signal informative.

The appeal of REINFORCE-based training is that it directly optimizes the metric you care about, bypassing the mismatch between cross-entropy training loss and evaluation metrics. The downside is that sampling-based gradient estimation is noisy and computationally expensive. You must sample complete sequences, evaluate them, and backpropagate through potentially long sequences. This is typically much slower than a single teacher-forced forward-backward pass.