Teacher Forcing: Training Seq2Seq with Ground Truth Context

When you train a sequence-to-sequence model to translate "The cat sat on the mat" into French, the decoder must generate one token at a time: first "Le", then "chat", then "était assis", and so on. At each step, the decoder needs to know what it generated previously. This raises an immediate question: during training, do you feed the decoder its own (potentially wrong) previous predictions, or do you hand it the correct tokens from the reference translation?

Teacher forcing answers this question by always providing the correct previous token during training. The name comes from the analogy of a strict teacher who corrects students at each step rather than letting errors compound. Instead of waiting for the student to figure out their mistakes through experience, the teacher intervenes immediately to put them back on track. The student never practices recovering from their own errors; they always move forward from the correct starting point.

This seemingly simple decision has enormous practical consequences. Teacher forcing makes training dramatically faster and more stable, but it also creates a subtle mismatch between how a model is trained and how it is used at inference time. Understanding this mismatch, called exposure bias, and the strategies developed to mitigate it is central to building reliable sequence generation systems. The history of these mitigations tracks some of the most important ideas in sequence modeling research over the past decade.

As we established in the Encoder-Decoder Framework chapter, seq2seq models process input through an encoder that produces a context vector, then use a decoder that generates output tokens one at a time. Teacher forcing is specifically about how the decoder receives its input at each time step during training. The choice matters deeply because it directly affects training stability, convergence speed, and final model quality, and because it determines whether the model ever learns to handle the kinds of mistakes it will inevitably make when running on its own at inference time.

To understand why teacher forcing was invented, consider what happens without it. A seq2seq decoder operates autoregressively: each output token depends on all previously generated tokens. During inference, this is unavoidable, since you do not know what the correct output is. You simply feed each generated token back as input for the next step.

During training, you have the correct target sequence. So you face a choice that is more consequential than it first appears.

Free-running (fully autoregressive) training feeds the decoder's own previous predictions as input at each step. If the decoder generates a wrong token early in the sequence, that error propagates forward. The next step receives a wrong input, which makes the next output more likely to be wrong, and so on. The model must learn to recover from its own mistakes without any external guidance.

Teacher forcing replaces the decoder's previous prediction with the correct target token. At step $t$, the decoder receives the ground-truth token $y_{t-1}$ rather than its own prediction $\hat{y}_{t-1}$. The model never has to deal with the cascading consequences of its own errors during training.

Why does cascading matter so much? Consider training a model to translate a twenty-word sentence. If the model makes an error at position 3 and that error shifts the decoder into an incorrect hidden state, then positions 4 through 20 all receive a corrupted context. The gradients flowing back through those positions all carry signal about how to respond to corrupted states rather than about how to correctly translate the language. The model spends most of its learning budget on a task it will never face in a fully-supervised setting: error recovery. This makes training both slow and unstable.

Teacher forcing eliminates this problem entirely. Every gradient update teaches the model about the correct conditional distribution conditioned on correct context. Learning is fast and the training signal is clean.

The mathematical framing clarifies what each approach optimizes. In standard language modeling, the loss at each step is the cross-entropy between the predicted distribution and the true next token. With teacher forcing, the loss at step $t$ is:

where:

• $y_t$: the true target token at step $t$
• $y_1, \ldots, y_{t-1}$: the ground-truth tokens at all previous steps
• $\mathbf{c}$: the encoder context vector (or set of context vectors with attention)
• $P(\hat{y}_t = y_t \mid \ldots)$: the decoder's predicted probability for the correct token

The total training loss sums over all time steps:

This is clean and efficient because every conditional probability is conditioned on correct context, making gradients stable and informative throughout the sequence. The gradients from step 20 flow back cleanly to earlier parts of the network without being corrupted by the effects of accumulated errors.

Without teacher forcing, the loss would instead be conditioned on the model's own predictions, which change as training progresses. This creates a non-stationary training signal that is much harder to optimize. The model is essentially trying to hit a moving target: the distribution it needs to learn from depends on its own current quality, so as the model improves, the training distribution shifts. This instability is one reason free-running training from scratch is rarely used for seq2seq models.

The mechanics of teacher forcing are straightforward once you see the decoder's input-output structure clearly.

In an encoder-decoder model, the decoder receives two inputs at each time step. The first is the previous token embedding, which is the recurrent input carrying information about what was generated just before. The second is the encoder's context information, which is either a single fixed hidden state passed from the encoder or, in models with attention, a dynamically weighted combination of encoder states. Teacher forcing changes only the first of these: how the "previous token" is determined.

With teacher forcing, the "previous token" input is always taken from the ground-truth target sequence, shifted right by one position. If the target sequence is $\langle\text{SOS}\rangle, y_1, y_2, \ldots, y_T, \langle\text{EOS}\rangle$, then:

• At step 1: decoder receives $\langle\text{SOS}\rangle$ as input, should predict $y_1$
• At step 2: decoder receives $y_1$ as input, should predict $y_2$
• At step $t$: decoder receives $y_{t-1}$ as input, should predict $y_t$
• At step $T+1$: decoder receives $y_T$ as input, should predict $\langle\text{EOS}\rangle$

The key insight is that the decoder sees the entire target sequence shifted one position to the right, so every conditional probability is conditioned on the true context rather than noisy predictions. In practice this is efficient to implement: the entire target sequence can be fed into the decoder in parallel (for transformer architectures) or in a tight loop where each teacher-forced input is a tensor slice from a pre-loaded batch of targets.

It is worth pausing on why this works so efficiently from a computational perspective. Because the teacher-forced inputs are all drawn from the target sequence (known ahead of time), you can precompute the entire sequence of decoder inputs before starting the forward pass. In RNN-based seq2seq models, this allows a tight training loop. In transformer-based models, it allows fully parallel processing of all decoder positions simultaneously, which is a key reason transformers can be trained so much faster than RNNs on modern hardware. Teacher forcing improves accuracy and is also a prerequisite for the computational efficiency that makes large-scale training feasible.

Transformers exploit teacher forcing more aggressively than RNNs. In an RNN decoder, even with teacher forcing, you must step through positions sequentially because each hidden state depends on the previous one. In a transformer decoder, the entire target sequence is fed in at once, and the attention mechanism computes all positions in parallel. The causal masking ensures that position $t$ can only attend to positions $1, \ldots, t-1$, which enforces the autoregressive property while preserving full parallelism during training.

This means transformer training processes all $T$ tokens simultaneously per layer pass, rather than requiring $T$ sequential RNN steps. The result is the dramatic training speedup that made models like BERT and GPT practical at scale. Teacher forcing is the mechanism that makes this parallel training coherent: without it, you would need to wait for each prediction before feeding the next input, collapsing back to sequential processing.

The consequence is that teacher forcing is architecturally necessary for the transformer's parallel training regime. You cannot train a transformer decoder without providing it the full right-shifted target sequence as input during the forward pass. The causal masking handles the autoregressive constraint, and the teacher-forced targets handle the input. Remove either of these and the transformer's training paradigm breaks down.

Teacher forcing accelerates training significantly, but it introduces a critical problem: the model is trained on a distribution that it never encounters during inference.

During training with teacher forcing, the decoder always receives correct tokens as input. The model learns the conditional distribution $P(y_t \mid y_1, \ldots, y_{t-1}, \mathbf{c})$ where the conditioning context is always drawn from the true sequence.

During inference (free-running decoding), the decoder receives its own previous predictions. The model encounters the distribution $P(y_t \mid \hat{y}_1, \ldots, \hat{y}_{t-1}, \mathbf{c})$ where the context consists of potentially imperfect predictions.

This mismatch is called exposure bias because the model is never "exposed" to its own errors during training. The consequence is that even small mistakes early in generation can derail the model: the first wrong token puts the decoder in a state it never encountered during training, which leads to another wrong token, which leads to another unfamiliar state, and so on. This compounding of errors is sometimes called the "snowball effect."

To develop strong intuition for why this matters, consider a concrete example. Suppose you are training a translation model, and in the target language, a particular grammatical construction requires a specific word order. The model has seen thousands of examples of this construction during training, always with perfect teacher-forced context. It has learned to produce the correct word order given correct input.

At inference time, the model generates the first word of this construction correctly, but then makes a small error in the second word. Now the third word must follow a malformed second word, a situation the model has never seen during training. Because this state is out-of-distribution, the model has no reliable behavior here. It might produce something locally plausible given the wrong second word, pushing the output further in the wrong direction, or it might generate something entirely unexpected because this input region is sparse in the model's learned distribution.

The severity of exposure bias depends on sequence length and output vocabulary size. Short sequences with limited vocabulary (like simple arithmetic operations) may show minimal degradation. Long sequences with large vocabularies (like paragraph-length text generation) can suffer substantially: a model that achieves near-perfect performance with teacher forcing might generate incoherent text when running freely. This explains why early seq2seq models often performed well on benchmark metrics computed with teacher-forced decoding but showed noticeably worse quality in real deployment, where no teacher is present.

To quantify the gap formally, define the decoder's true data distribution at step $t$ as $p_{\text{data}}(y_t \mid y_{<t})$ and the model's prediction as $p_{\theta}(y_t \mid y_{<t})$. At training step $t$, the input is $y_{<t} \sim p_{\text{data}}$ (teacher forcing), but at inference step $t$, the input is $\hat{y}_{<t} \sim p_{\theta}$. As sequences grow longer, $\hat{y}_{<t}$ diverges increasingly from $y_{<t}$, amplifying the distribution shift. The compounding nature of this divergence is what makes exposure bias worse for longer sequences: each step's imperfection contributes to pushing the inference-time trajectory further from the training-time trajectory.

One way to measure exposure bias directly is to evaluate a model using two different decoding strategies and compare. The first strategy is teacher-forced decoding, where even at "inference time" you provide correct previous tokens. This measures the model's upper-bound performance in the perfect-information setting. The second strategy is free-running decoding, where the model must use its own previous predictions. The gap between these two metrics quantifies the severity of exposure bias for that model.

Large gaps indicate the model has learned very precisely calibrated conditionals over correct sequences but struggles to generalize to imperfect sequences. Smaller gaps suggest the model is either more robust to distributional shift, or that the task itself has less compounding error because the vocabulary is small or sequences are short. Monitoring this gap during training can tell you whether a particular mitigation strategy is working, and whether you need to invest in more aggressive exposure bias reduction.

Bengio et al. (2015) proposed scheduled sampling as a curriculum learning approach to address exposure bias. The core idea is to gradually transition from teacher forcing to free-running training during the training process, so the model learns to handle its own outputs before deployment.

In scheduled sampling, at each training step you flip a biased coin. With probability $\epsilon_t$ (the "scheduled" probability), you feed the true previous token as input (teacher forcing mode). With probability $1 - \epsilon_t$, you feed the model's own previous prediction (free-running mode).

where:

• $\epsilon_t$: the teacher forcing probability at training step $t$, which decays over time
• $y_{t-1}$: the ground-truth token at position $t-1$
• $\hat{y}_{t-1}$: the model's own predicted token (argmax or sampled) at position $t-1$

The scheduled part comes from how $\epsilon_t$ is decayed over training. Three common schedules are:

Linear decay: $\epsilon_i = \max(\epsilon_{\min}, k - i \cdot c)$, where $i$ is the training iteration and $c$ controls decay speed. This reduces teacher forcing at a constant rate, giving a predictable and easy-to-tune schedule. The $\epsilon_{\min}$ floor prevents teacher forcing from going all the way to zero, which helps maintain some training stability even late in training.

Exponential decay: $\epsilon_i = k^i$, where $k < 1$. This decays quickly at first, then slows as it approaches zero. Because the schedule starts steep, the model is transitioned away from teacher forcing aggressively in early iterations and then receives a more gradual nudge in later iterations. This is suitable when you want the model to encounter its own errors early in training.

Inverse sigmoid decay: $\epsilon_i = k / (k + \exp(i / k))$. This produces an S-shaped schedule, staying near 1.0 for a while before declining steadily to near 0.0. The effect is that the model receives nearly full teacher forcing during its early learning phase, which is when the basic conditional distributions are being established. Then, once the model has developed some baseline capability, the schedule allows the teacher forcing ratio to drop, progressively exposing the model to its own outputs.

The intuition behind the curriculum approach is that a model needs to first learn the basic conditional distributions (what tokens follow what) before it can learn to recover from mistakes. Starting with pure teacher forcing provides stable gradients for this initial learning. Then, as training progresses, introducing the model's own outputs teaches it to remain coherent even when its context is imperfect.

The choice of schedule affects more than just training speed. A schedule that drops $\epsilon$ too quickly will destabilize training before the model has enough capability to learn from free-running feedback. A schedule that drops $\epsilon$ too slowly may not provide enough exposure to the model's own errors before training ends, leaving significant residual exposure bias. In practice, the inverse sigmoid schedule is often recommended because it provides the best of both regimes: a stable initial phase and a controlled transition.

Scheduled sampling is an instance of a broader idea called curriculum learning, where the training distribution is intentionally structured to present easier examples first, then progressively harder ones. The "easy" version here is teacher forcing, where the model never needs to recover from mistakes. The "hard" version is free-running decoding, where the model must handle all the consequences of its own imperfect outputs.

This curriculum approach mirrors how humans learn to perform complex sequential tasks. A student learning to write learns first by copying model sentences, which is analogous to teacher forcing. Then they move to writing with occasional corrections, where the teacher steps in when errors would compound badly. Finally they write independently, handling their own mistakes as they arise. Direct exposure to the fully independent task from the start would be overwhelming because the feedback signal is too noisy to learn from efficiently; the curriculum manages the complexity gradient.

Curriculum learning in general has been shown to accelerate training across many domains beyond seq2seq models. The key principle is that the learning signal should be appropriately calibrated to the student's current capability. When the student is weak, easy examples with clean feedback are most informative. When the student is stronger, harder examples that push the boundaries of what they can handle accelerate further learning. Scheduled sampling applies this principle specifically to the question of input quality in sequential decoding.

One subtle point is that scheduled sampling is not a perfect solution. It can introduce inconsistency: at any given training step, some positions in the sequence use teacher forcing while others use free-running predictions. This creates a hybrid conditioning context that the model must handle, which can sometimes make learning harder rather than easier. The model sees sequences where some tokens are correct (from the ground truth) and adjacent tokens are potentially wrong (from the model's own predictions), creating conditioning contexts that are neither fully correct nor fully free-running. Handling this mixture requires the model to develop a kind of robustness to partial context errors, which may require more model capacity or more careful training management.

A further subtlety arises when you consider gradient computation. With pure teacher forcing, gradients flow cleanly through fixed inputs. With scheduled sampling, the model's own predictions are used as inputs, and those predictions are the output of differentiable operations. In principle, you could backpropagate through the sampling operation itself to train the model end-to-end. In practice, most implementations treat the sampled tokens as fixed and do not backpropagate through the sampling step, because the argmax (or categorical sample) is not differentiable. This means scheduled sampling does not fully exploit the information available in the model's own predictions, leaving some optimization potential on the table.

An alternative approach, proposed by Lamb et al. (2016), is professor forcing. Rather than mixing teacher-forced and free-running training at the input level, professor forcing uses an adversarial training objective to align the model's hidden state distributions between the two modes.

The approach trains a discriminator to distinguish between two types of hidden state trajectories. The first type consists of hidden states produced by teacher forcing, where inputs are ground-truth tokens and the decoder's internal representations reflect a clean, correct sequence of states. The second type consists of hidden states produced by free-running decoding, where inputs are the model's own predictions and the representations reflect the actual trajectory the model would follow at inference time.

The generator, which is the seq2seq model itself, then tries to fool the discriminator by producing hidden states under free-running conditions that look statistically indistinguishable from teacher-forced hidden states. This adversarial signal pushes the free-running trajectory to stay close to the teacher-forced trajectory in representation space, not just in output token space.

Why is this a different and potentially more powerful approach than scheduled sampling? Scheduled sampling addresses the problem at the input level: it teaches the model to produce reasonable outputs when given its own previous output as input. But it does not directly constrain how the model's hidden states evolve during free-running. The hidden states are what the model uses to compute its output distributions, so if the hidden states under free-running diverge substantially from those under teacher forcing, the model's behavior will still degrade even if the individual input tokens are similar.

Professor forcing adds an explicit objective that the model's internal representations should remain consistent regardless of whether it is operating in teacher-forced or free-running mode. This forces a kind of representation-level robustness: the model must learn to maintain useful hidden state structure even when its inputs are its own potentially imperfect predictions. The discriminator provides a direct signal about whether the internal states are deviating from the teacher-forced distribution, which is something that cross-entropy loss alone cannot measure.

The practical limitation of professor forcing is that it requires training an adversarial discriminator alongside the main model. Adversarial training is notoriously sensitive to hyperparameter settings and can be difficult to stabilize. The discriminator must be capable enough to detect distribution differences but not so powerful that it makes the generator's task impossible. Despite these challenges, professor forcing demonstrated that addressing exposure bias at the representation level rather than just the input level is a valid and effective approach, and it inspired later work on representation regularization for sequence models.

A related training paradigm called data as demonstrator (DaD) takes yet another approach. Instead of using only ground-truth tokens as teacher inputs or mixing in the model's own predictions, it trains the model by letting it observe token sequences generated by a high-quality reference model or oracle.

In the DaD framework, the process works as follows. First, a teacher model (which may be a larger, pretrained model or a well-performing domain expert) generates multiple reference sequences for each input. Then, the student model is trained to imitate the teacher's token choices. The distinction from standard teacher forcing is that the teacher's own generation errors are included, exposing the student to realistic decoder trajectories that reflect what a competent but not perfect model produces.

The advantage is that the student learns from a realistic distribution of outputs rather than from perfectly correct sequences that may never appear during inference. Human reference translations, for instance, are grammatically correct but may use different word choices than the model would naturally produce. If the model sees only human references during training, it learns to condition on human-style outputs. When it runs freely, it produces model-style outputs, creating a mismatch. A teacher model that is similar in kind to the student (a seq2seq neural network generating plausible but not perfect outputs) provides conditioning contexts that more closely resemble what the student will experience during inference.

The disadvantage is the need for a capable teacher model, which creates a bootstrapping challenge in the early stages of training. You need a good model to generate the training data for a good model. In practice, this is often handled by using a pretrained model from a related task or a previous training run, then iteratively improving both the teacher and student through multiple rounds of training.

A fundamentally different approach to bridging the train-test gap is to directly optimize the evaluation metric using reinforcement learning. The REINFORCE algorithm (Williams, 1992) applied to seq2seq models treats the decoder as a policy that takes actions (token choices) to maximize a reward signal.

The motivation for this approach is straightforward but important. Cross-entropy loss with teacher forcing measures how well the model predicts each token given correct context. But the thing you care about in machine translation or summarization is not per-token prediction accuracy; it is the quality of the complete output sequence as measured by metrics like BLEU (for translation) or ROUGE (for summarization). These metrics are not decomposable step by step; they measure properties of the entire generated sequence. Teacher forcing cannot directly optimize them.

The key idea in REINFORCE-based training is to define the reward $R(\hat{y}, y)$ as the task-specific evaluation score between the generated sequence $\hat{y}$ and the ground-truth sequence $y$. For machine translation, this could be BLEU score. For summarization, this could be ROUGE. For dialogue, it could be a human preference score.

The policy gradient update for the model parameters $\theta$ is:

where:

• $\hat{y} \sim p_\theta$: a complete output sequence sampled from the model
• $R(\hat{y}, y)$: the reward signal comparing generated output to ground truth
• $\nabla_\theta \log p_\theta(\hat{y} \mid \mathbf{x})$: the gradient of the log probability of the sampled sequence

The intuition is: sample a complete output sequence, evaluate how good it was using the task metric (the reward), then update parameters to make good sequences more likely and bad sequences less likely. This directly aligns the training objective with the evaluation objective, which is the fundamental appeal of the approach.

However, REINFORCE alone is unstable in practice due to high variance in the gradient estimates. Consider what happens when you sample a sequence and measure its BLEU score. The BLEU score of a single sentence is noisy: a sequence might score low due to random word choice differences from the reference, not fundamental quality issues. The gradient is weighted by this noisy reward, so the updates are highly variable. With pure REINFORCE, training often fails to converge or converges to poor solutions.

Rennie et al. (2017) introduced the self-critical training sequence (SCST) approach, which uses the model's own greedy decoding output as a baseline to reduce variance. Rather than using the raw reward, the update uses the advantage: how much better or worse was the sampled sequence compared to the greedy sequence?

where $\bar{y}$ is the greedy decoding output. This baseline reduces variance because the gradient is non-zero only when the sampled sequence is better or worse than the greedy output. If both sequences receive the same reward, no update is applied. This means the model is only updated when sampling discovers something the greedy decoding missed (positive advantage) or when it stumbles into something worse (negative advantage). The updates carry meaningful signal rather than noise.

The self-critical baseline has an elegant property: it is computed using the same model being trained, so it adapts automatically as the model improves. As the greedy output quality increases over training, the baseline rises, maintaining a useful performance gap that keeps the learning signal informative.

The appeal of REINFORCE-based training is that it directly optimizes the metric you care about, bypassing the mismatch between cross-entropy training loss and evaluation metrics. The downside is that sampling-based gradient estimation is noisy and computationally expensive. You must sample complete sequences, evaluate them, and backpropagate through potentially long sequences. This is typically much slower than a single teacher-forced forward-backward pass.

In modern practice, cross-entropy loss (teacher forcing) and REINFORCE-style rewards are rarely used in isolation. Instead, they are combined in a mixed training objective that exploits the strengths of both approaches. The standard formulation is:

where:

• $\lambda$: the mixing coefficient, typically set between 0.9 and 0.99 to keep training stable
• $\mathcal{L}_{\text{CE}}$: the cross-entropy (teacher forcing) loss
• $\mathcal{L}_{\text{RL}}$: the policy gradient (REINFORCE) loss

The high value of $\lambda$ reflects the practical wisdom that cross-entropy loss provides essential regularization. Without it, the RL component alone tends to overfit to the specific evaluation metric, sometimes in degenerate ways. Models trained only with BLEU score have been known to learn to repeat high-frequency phrases that boost BLEU without improving actual translation quality. The cross-entropy component keeps the model's predicted distribution over tokens close to the true distribution, which prevents degenerate reward-hacking behavior.

A common training recipe in industrial NMT systems is to first train to convergence with pure teacher forcing, then switch to the mixed objective for a limited number of additional epochs. The rationale is that teacher forcing is far more efficient during the initial phase, since it provides a clean and informative gradient signal at every step. Once the model has developed strong basic competence, the RL fine-tuning phase can safely optimize the task metric with reasonable stability.

This two-stage approach also addresses the cold-start problem for RL training. If you apply REINFORCE from the beginning, the model's initial random outputs receive very low rewards, and the gradient variance is enormous because the baseline (the greedy output) is also terrible. There is very little useful signal for the model to learn from. Starting from a strong teacher-forced checkpoint means the model already produces reasonable outputs, the rewards are in a useful range, and the advantage signal (difference between sampled and greedy rewards) is informative.

Each training strategy makes different tradeoffs between training stability, inference quality, and computational cost. Understanding these tradeoffs helps you choose the right approach for a given task and budget.

Pure teacher forcing is the most common approach and the default for most seq2seq systems. It provides fast training with stable gradients because every training step conditions on correct context. Performance with teacher-forced evaluation is strong, but there is a significant exposure bias at inference time for long sequences. For many tasks, especially with modern attention-based models, the exposure bias is manageable and teacher forcing alone produces competitive results.

Free-running training (no teacher forcing from the start) is practically infeasible for most tasks because of cascading errors. The training signal is noisy and unstable, and convergence either fails or is extremely slow. Free-running training is not used as a primary training approach; it is primarily a theoretical baseline that illustrates why teacher forcing was needed in the first place.

Scheduled sampling represents a practical middle ground. Training speed is intermediate between teacher forcing and free-running, and the model progressively learns to handle its own outputs. The schedule hyperparameters require tuning, and the inconsistency from mixed conditioning contexts can sometimes slow learning or require more model capacity. But for tasks with significant exposure bias issues, scheduled sampling reliably reduces the train-test gap.

REINFORCE and SCST directly optimize the task metric, which is their primary advantage. But they suffer from high gradient variance, require sampling complete sequences during training (computationally expensive), and can be unstable without careful hyperparameter management. These approaches are most commonly used as a fine-tuning step after teacher forcing pretraining, not as the primary training method.

In practice, the choice of training strategy also depends on the architecture. Transformer-based seq2seq models tend to show less exposure bias than RNN-based models, probably because their attention mechanisms allow them to condition flexibly on all previous positions rather than relying solely on a compressed hidden state that may carry errors. For transformer models, pure teacher forcing often suffices. For RNN models on long sequences, scheduled sampling or RL fine-tuning may be necessary to achieve good free-running performance.

Let's implement teacher forcing and compare it to scheduled sampling using a simple character-level seq2seq model. This will make the mechanics concrete and reveal the exposure bias effect directly.

First, we install the necessary libraries and set up the basic data.

The exact sequence match accuracy under free-running decoding reveals the practical impact of exposure bias. The task requires the model to get every single token right to score a match, making it a stringent measure of inference quality. The scheduled sampling model, having practiced handling its own outputs during training, tends to produce more consistent complete sequences at inference time, especially for sequences where an early error would cascade into multiple downstream mistakes.

Understanding the key hyperparameters for teacher forcing and scheduled sampling helps you apply these techniques effectively:

• teacher_forcing_ratio: Probability of using the ground-truth token as decoder input. Set to 1.0 for pure teacher forcing, 0.0 for free-running. Scheduled sampling decays this value during training. The initial value should almost always start at 1.0 to ensure stable early training.
• schedule type: The decay function controlling how teacher forcing probability decreases. Linear decay is simple and predictable; inverse sigmoid provides a more controlled transition that preserves the stable learning phase.
• epsilon_min: The floor value for scheduled sampling probability. Keeping this above 0 ensures some ground-truth guidance persists even late in training, which can stabilize learning. A common choice is 0.0 to 0.2 depending on task difficulty.
• decay rate: Controls how quickly teacher forcing probability falls. Faster decay exposes the model to its own errors sooner, which can reduce exposure bias but also destabilizes early training if the model has not yet learned reliable basic patterns.

To make the mechanics completely concrete, let's trace through a single decoding step with and without teacher forcing.

Suppose the input sequence is [5, 3, 8, 4, 6, 7] and the correct reversed output is [7, 6, 4, 8, 3, 5]. The target sequence with special tokens is [SOS, 7, 6, 4, 8, 3, 5, EOS].

With teacher forcing, at step 3 (predicting the third output token), the decoder receives:

• Encoder context: the hidden state from encoding [5, 3, 8, 4, 6, 7]
• Decoder input: 6 (the ground-truth second token), regardless of what the model predicted at step 2

The model might have predicted 9 at step 2 (an error), but teacher forcing ignores this and provides 6 anyway. The gradient at step 3 correctly attributes blame for any error at step 3 to the model's handling of the correct input 6, not to a corrupted context.

With free-running decoding at inference time, step 3 receives:

• Encoder context: same hidden state from encoding [5, 3, 8, 4, 6, 7]
• Decoder input: 9 (the model's incorrect prediction from step 2)

The model must now condition on 9, which is not part of the original sequence, and figure out what the third reversed token should be. Because 9 was never a valid step-2 output in this task, this is an out-of-distribution input. The model's output at step 3 will reflect its behavior in this novel state, which may bear little resemblance to what it learned from correctly-conditioned training examples.

This single example illustrates the core of exposure bias: the model's training distribution (conditioned on correct tokens) and inference distribution (conditioned on its own predictions) diverge every time it makes a mistake. For a length-8 sequence, a single error at step 2 means steps 3 through 7 are all operating in unfamiliar territory. Scheduled sampling directly addresses this by ensuring that some training steps experience this exact kind of imperfect conditioning, teaching the model to recover from it.

Teacher forcing was a critical enabler for practical seq2seq training, but it comes with real tradeoffs that are worth understanding deeply before applying it.

The exposure bias problem has measurable practical effects. In neural machine translation benchmarks, models trained with pure teacher forcing often show a significant drop in BLEU score when the source sentences are longer or when beam search is used with a larger beam width. The model, having never encountered its own errors during training, lacks the learned ability to course-correct when decoding goes off track. This effect is most pronounced for languages with long-range dependencies or flexible word order, where an error early in generation can propagate for many tokens before the sentence structure constrains the output back toward coherence.

The gap between teacher-forced training performance and free-running inference performance led researchers in the 2015-2018 period to develop the mitigation strategies described in this chapter. Scheduled sampling reduced the gap but did not eliminate it. Professor forcing added a representation-level constraint but required adversarial training with all its instabilities. REINFORCE-based approaches directly optimized task metrics but were expensive and noisy. None of these approaches fully solved the problem; each traded one set of difficulties for another.

Scheduled sampling alleviates exposure bias but does not fully resolve it. The mixing of teacher-forced and free-running tokens within the same training sequence creates a hybrid conditioning context that can confuse the model. Theoretically, the model should learn to handle both modes, but in practice the inconsistency sometimes hurts learning efficiency. Some researchers have found that scheduled sampling requires careful hyperparameter tuning and can be sensitive to the choice of schedule. The inconsistency problem is particularly sharp at the boundary between teacher-forced and free-running positions: a sequence where positions 1-4 are teacher-forced and position 5 is free-running presents the model with an unusual conditioning context that it would never encounter during pure inference.

A practical observation from industrial NMT systems is that even with exposure bias, teacher forcing produces competitive models when paired with beam search at inference time. Beam search effectively hedges against early mistakes by maintaining multiple hypotheses simultaneously, re-evaluating them as the sequence grows and discarding branches that fall too far behind the leading candidates. The next chapter covers beam search in detail and explores how its width and length normalization interact with the training strategy. The interaction is significant: models trained with teacher forcing tend to respond well to beam search because they have learned sharp conditional distributions over correct sequences, and beam search can exploit these sharp distributions to find good completions even when early steps are uncertain.

The advent of transformer-based seq2seq models has somewhat shifted the conversation about exposure bias. Transformers trained with masked self-attention use the same teacher forcing paradigm, and they have proven substantially more robust to exposure bias than RNN-based models. This robustness likely comes from several factors. Their attention mechanisms allow them to condition on all previous positions simultaneously, which reduces the compounding error effect: a wrong token at position $t$ propagates its influence through attention weights, but the weights to other correct tokens can compensate. Their large capacity allows them to learn more robust conditional distributions that generalize better across the small distribution shift from teacher-forced to free-running contexts. And their parallel training (which requires teacher forcing) allows them to be trained at scales where the sheer volume of diverse training examples reduces the impact of any particular conditioning pattern being out-of-distribution at inference time.

Nevertheless, the fundamental train-test mismatch remains in all teacher-forced models, transformer or otherwise. Understanding it is essential for diagnosing generation failures. When a model produces incoherent long-form text despite good perplexity on held-out data, exposure bias is often the culprit. When a model succeeds on short test inputs but fails on longer ones, the compounding nature of exposure bias is a natural suspect. When RL fine-tuning dramatically improves generation quality without changing the model's perplexity, it is often because the RL phase is teaching the model to handle the imperfect contexts it will encounter at inference time.

For practitioners building production sequence generation systems, the most common recipe is to train with pure teacher forcing to convergence, then apply a brief RL fine-tuning phase using a mixed cross-entropy and task-metric objective. This strategy combines the efficiency of teacher forcing with the inference-time optimization of reinforcement learning, and it has been validated across machine translation, abstractive summarization, image captioning, and dialogue systems.

Teacher forcing is a training technique that provides the correct previous token to the decoder at each step, rather than feeding back the model's own predictions. The key takeaways from this chapter are:

• Efficiency: Teacher forcing provides stable gradients and fast convergence because every training step conditions on correct context. It also enables fully parallel training in transformer architectures, which is essential for large-scale training.
• Exposure bias: The train-test mismatch arises because the model is never exposed to its own errors during training, but must handle them at inference time. This causes compounding errors during free-running decoding, especially for long sequences.
• Scheduled sampling: A curriculum learning approach that gradually transitions from teacher forcing to free-running by decaying the teacher forcing probability over training. The inverse sigmoid schedule is often the most effective because it preserves a stable initial learning phase.
• Professor forcing: An adversarial approach that aligns the model's hidden state distributions between teacher-forced and free-running modes, addressing exposure bias at the representation level rather than just the input level.
• REINFORCE and SCST: Policy gradient approaches that directly optimize task metrics. Self-critical sequence training uses the model's own greedy output as a variance-reducing baseline.
• Practical recipe: Most modern systems pretrain with teacher forcing for efficiency, then fine-tune with a mixed objective combining cross-entropy and reinforcement learning. This two-stage approach combines training efficiency with inference-time quality.
• Transformer robustness: Transformer-based models show less exposure bias than RNN-based models, likely due to their attention mechanisms and large training scales, but the fundamental train-test mismatch persists.

In the next chapter, we will look at beam search, the standard decoding algorithm used at inference time. Beam search partially compensates for exposure bias by maintaining multiple candidate hypotheses throughout decoding, but it introduces its own set of tradeoffs around beam width and length normalization.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about teacher forcing and exposure bias in sequence-to-sequence training.