T5 Pre-training › Span Corruption: The Core Objective

Span corruption turns input text into a denoising task, fundamentally reshaping how the model learns to understand language. The core idea is straightforward: rather than asking a model to fill in individual blanks scattered throughout a sentence, we ask it to reconstruct entire missing phrases. This seemingly simple change has measurable implications for what the model must learn.

Consider the difference between predicting a single missing word versus predicting a missing phrase. When filling in a single blank, a model can often succeed by identifying the grammatical category needed (noun, verb, adjective) and selecting a plausible word from that category. However, when an entire phrase is missing, the model must generate multiple words that fit the surrounding context and cohere with each other. The phrase "large amounts" requires understanding that these two words naturally pair together, that they describe a quantity, and that they grammatically connect "require" to "of training data." This multi-token challenge forces the model to develop richer internal representations of language structure.

The procedure works in three distinct stages: selecting spans to corrupt, replacing them with sentinel tokens, and training the model to reconstruct the original spans. Each stage involves design decisions from extensive testing.

The algorithm begins by deciding which tokens to corrupt, a process that must balance several competing concerns. We want enough corruption to give real training signal. If we corrupt too little, each training example teaches the model relatively little. However, we also want the task to remain solvable. If we corrupt too much, the model cannot reasonably infer what was removed. T5 uses a corruption rate of 15%, meaning roughly 15% of tokens in each sequence will be masked. This particular rate emerged from experiments as a good balance that gives strong learning signal while keeping the reconstruction task feasible.

However, rather than selecting tokens independently like BERT does, T5 groups consecutive tokens into spans. This grouping decision is more than aesthetic. It fundamentally changes the nature of what the model learns. When BERT masks the word "fox" in "The quick brown fox jumps," the model learns that an animal noun fits this context. When T5 masks "brown fox" as a single span, the model must learn that an animal noun fits here, that adjective-noun pairs describing animals have particular patterns, that "brown" and "fox" co-occur naturally, and that the resulting phrase must connect smoothly with both "quick" before it and "jumps" after it.

The span lengths follow a geometric distribution with a mean of 3 tokens. This means short spans are most common, but occasionally longer spans appear. Why a geometric distribution? This distribution has the useful property that span lengths are memoryless: given that a span has already reached length $k$, the probability it continues for another token remains constant. This memoryless property simplifies the sampling procedure and ensures a natural spread of span lengths without complicated bookkeeping.

For a geometric distribution with mean $\mu = 3$, the probability of a span having exactly length $k$ is:

$$P(\text{length} = k) = (1-p)^{k-1} \cdot p$$

where:

• $k$: the span length in tokens (must be at least 1)
• $p$: the probability parameter of the geometric distribution, set to $p = 1/\mu = 1/3$
• $(1-p)^{k-1}$: the probability of "failing" to stop for the first $k-1$ tokens
• $p$: the probability of "stopping" at exactly the $k$-th token

To build intuition for this formula, imagine flipping a biased coin for each token in the span. The coin has probability $p = 1/3$ of coming up "stop" and probability $1-p = 2/3$ of coming up "continue." A span of length 1 occurs when we flip "stop" immediately on the first token, which happens with probability $p = 1/3$. A span of length 2 occurs when we flip "continue" on the first token (probability $2/3$) and then "stop" on the second (probability $1/3$), giving $(2/3) \cdot (1/3) = 2/9 \approx 0.22$. A span of length 3 requires "continue, continue, stop," which is $(2/3)^2 \cdot (1/3) = 4/27 \approx 0.15$.

With $p = 1/3$, this gives us probabilities of approximately 33% for length 1, 22% for length 2, 15% for length 3, and so on. The distribution decays exponentially. This keeps very long spans are rare but not impossible. This mix of span lengths exposes the model to a variety of reconstruction challenges: sometimes it must predict just a single word, sometimes a short phrase, and occasionally an extended expression.

When T5 corrupts a span, it replaces the entire span with a single sentinel token, regardless of how many tokens the span contained. The first corrupted span becomes <extra_id_0>, the second becomes <extra_id_1>, and so forth. This compression is important: it means the corrupted input is materially shorter than the original, making training more efficient.

The naming convention "extra_id" shows that these tokens exist outside the normal vocabulary. They carry no inherent meaning and exist purely as markers. Think of them as numbered placeholders, similar to how a librarian might place numbered bookmarks where removed pages once sat. Each sentinel uniquely identifies which span was removed from that position, letting the model to precisely reconstruct what belongs where.

This design choice has an elegant efficiency benefit. If a span contains four tokens and we replace it with a single sentinel, we've reduced the input length by three tokens. Across an entire sequence with multiple corrupted spans, these savings accumulate substantially. A 512-token input might shrink to 460 tokens after corruption, letting the encoder to process more information per unit of computation. This efficiency gain compounds across millions of training examples.

The target sequence contains the sentinel tokens followed by the spans they replaced, in order. This creates a clear mapping between placeholders and their corresponding content. When the model sees <extra_id_0> in the encoder input, it knows that somewhere in the target sequence, <extra_id_0> will be followed by the exact tokens that belong in that position. The sentinel acts as both a placeholder in the input and a delimiter in the output.

Let's trace through the complete transformation with a concrete example, examining each step to build intuition for how span corruption restructures text into a training signal. Consider this input sentence:

"The quick brown fox jumps over the lazy dog and runs away."

Suppose the span corruption procedure selects two spans: "brown fox" (positions 2-3) and "lazy dog" (positions 7-8). The transformation produces:

Corrupted Input:

"The quick <extra_id_0> jumps over the <extra_id_1> and runs away."

Target Output:

"<extra_id_0> brown fox <extra_id_1> lazy dog"

To understand why this format is so effective, let's examine what the model must learn to successfully complete this task. Looking at the corrupted input, the model sees that something is missing between "quick" and "jumps." This something immediately follows "quick" and is modified by it. The model must infer that this is likely an animal (because animals jump) and that "quick" suggests it's being described with adjectives. The surrounding context of a sentence about jumping and running further constrains the possibilities.

Notice several key properties of this transformation:

1. The input is shorter than the original (sentinel tokens replace multi-token spans)
2. Each sentinel appears exactly once in the input and once in the target
3. The target contains only the corrupted spans, not the full original text
4. Sentinels in the target act as delimiters separating the reconstructed spans

This design means the model predicts far fewer tokens than the original sequence length, making training computationally efficient while still requiring deep understanding of context. The model doesn't waste computation predicting tokens that weren't corrupted. It focuses its predictive capacity entirely on the challenging reconstruction task.

The one-to-one correspondence between sentinels in input and target creates an elegant bookkeeping system. During generation, when the model outputs <extra_id_0>, it signals that the following tokens should fill the position marked by <extra_id_0> in the input. When it outputs <extra_id_1>, it transitions to filling the next gap. This structure allows the decoder to generate all missing content in a single autoregressive pass, without needing special position-tracking mechanisms.