Span Corruption › Worked Example: Tracing a Complete Training Sample

Before moving to the T5 training setup, it helps to trace a single training example from raw text all the way to loss computation. This concrete walk-through ties together every component we have built.

Suppose we have the sentence: "Researchers at Google trained the model on a large text corpus."

Step 1: Tokenization. We split this into 14 word tokens for simplicity (in practice, T5 uses SentencePiece subword tokenization, but the logic is identical):

That gives us $n = 12$ tokens.

Step 2: Span selection. With $r = 0.15$ and $\mu = 3$, we expect to corrupt $0.15 \times 12 \approx 2$ tokens spread across $2/3 \approx 1$ span. Let's say the sampler selects tokens 2 and 3 (indices 2-3, a span of length 2 covering "Google trained"):

Step 3: Building the corrupted input. Token positions 0-1 pass through unchanged. Positions 2-3 collapse to <extra_id_0>. Positions 4-11 pass through unchanged:

The input has 11 tokens, down from 12.

Step 4: Building the target sequence. The target begins with the sentinel, followed by the corrupted span's tokens:

Step 5: Encoder processing. The encoder receives all 11 input tokens and produces a hidden state matrix $H_{\text{enc}}$ of shape $11 \times d_{\text{model}}$. The hidden state at the <extra_id_0> position encodes everything the model can infer about the missing span from the surrounding context: "Researchers at ??? the model on a large text corpus" strongly implies that the missing words are a proper noun (an organization) followed by a verb. The encoder's bidirectional attention can draw on all 11 positions simultaneously.

Step 6: Decoder generation. The decoder starts with <extra_id_0> as its first input token, attending to all encoder states. It produces a probability distribution over the vocabulary and assigns high probability to "Google" (a proper noun that fits the context). "Google" is fed as input at the next step, and the decoder predicts "trained." Then the decoder sees "trained" and must decide what comes next: since the target ends here, the correct output is the end-of-sequence token (or in T5's format, simply no more tokens for this sentinel).

Step 7: Loss computation. The cross-entropy loss sums over the three target positions:

$$\mathcal{L} = -\log P(\texttt{<extra\_id\_0>} \mid \texttt{<extra\_id\_0>}, H_{\text{enc}}) - \log P(\texttt{Google} \mid \texttt{<extra\_id\_0>}, H_{\text{enc}}) - \log P(\texttt{trained} \mid \texttt{<extra\_id\_0>}, \texttt{Google}, H_{\text{enc}})$$

In practice the first term is trivially minimized since the target literally begins with the sentinel. The meaningful learning happens in the second and third terms, where the model must predict the actual missing words from context.

The main point from this trace is how naturally the task matches real generation. The model learns: "Given a partially visible sentence with a marked gap, produce the words that fill the gap." At inference time on real tasks like summarization or translation, the model does essentially the same thing: given an input (source text or passage), produce an output (summary or translation). Span corruption during pretraining directly practices this input-to-output mapping, which is why T5 transfers so effectively to later generation tasks.