Teacher Forcing › Worked Example: Tracing a Decoding Step

To make the mechanics completely concrete, let's trace through a single decoding step with and without teacher forcing.

Suppose the input sequence is [5, 3, 8, 4, 6, 7] and the correct reversed output is [7, 6, 4, 8, 3, 5]. The target sequence with special tokens is [SOS, 7, 6, 4, 8, 3, 5, EOS].

With teacher forcing, at step 3 (predicting the third output token), the decoder receives:

• Encoder context: the hidden state from encoding [5, 3, 8, 4, 6, 7]
• Decoder input: 6 (the ground-truth second token), regardless of what the model predicted at step 2

The model might have predicted 9 at step 2 (an error), but teacher forcing ignores this and provides 6 anyway. The gradient at step 3 correctly attributes blame for any error at step 3 to the model's handling of the correct input 6, not to a corrupted context.

With free-running decoding at inference time, step 3 receives:

• Encoder context: same hidden state from encoding [5, 3, 8, 4, 6, 7]
• Decoder input: 9 (the model's incorrect prediction from step 2)

The model must now condition on 9, which is not part of the original sequence, and figure out what the third reversed token should be. Because 9 was never a valid step-2 output in this task, this is an out-of-distribution input. The model's output at step 3 will reflect its behavior in this novel state, which may bear little resemblance to what it learned from correctly-conditioned training examples.

This single example illustrates the core of exposure bias: the model's training distribution (conditioned on correct tokens) and inference distribution (conditioned on its own predictions) diverge every time it makes a mistake. For a length-8 sequence, a single error at step 2 means steps 3 through 7 are all operating in unfamiliar territory. Scheduled sampling directly addresses this by ensuring that some training steps experience this exact kind of imperfect conditioning, teaching the model to recover from it.