T5 Pre-training › Mathematical Formulation

Having developed intuition for span corruption through examples, we can now formalize the objective mathematically. This formalization reveals the precise learning signal the model receives and connects span corruption to the broader family of sequence-to-sequence training objectives.

The span corruption objective can be formalized as follows. Given an input sequence $\mathbf{x} = (x_1, x_2, \ldots, x_n)$, we:

1. Sample span start positions and lengths according to the corruption parameters
2. Replace each span with a unique sentinel token to produce $\mathbf{x}_{\text{corrupted}}$
3. Construct target $\mathbf{y}$ by concatenating sentinels with their corresponding spans

The model is trained to maximize:

$$\mathcal{L} = \sum_{t=1}^{|\mathbf{y}|} \log P(y_t \mid y_{<t}, \mathbf{x}_{\text{corrupted}}; \theta)$$

where:

• $\mathcal{L}$: the log-likelihood objective to maximize (equivalently, minimizing negative log-likelihood as the loss)
• $\mathbf{y}$: the target sequence containing sentinels and their corresponding original spans
• $|\mathbf{y}|$: the length of the target sequence
• $y_t$: the $t$-th token in the target sequence
• $y_{<t}$: all target tokens before position $t$ (the autoregressive context)
• $\mathbf{x}_{\text{corrupted}}$: the input sequence with spans replaced by sentinel tokens
• $\theta$: the model parameters (encoder and decoder weights)
• $P(y_t \mid y_{<t}, \mathbf{x}_{\text{corrupted}}; \theta)$: the probability the model assigns to token $y_t$ given the corrupted input and previous target tokens

This formulation deserves careful unpacking. The summation runs over every token in the target sequence, meaning the model receives a learning signal for each prediction it makes. The conditioning on $\mathbf{x}_{\text{corrupted}}$ indicates that the encoder has processed the corrupted input and produced contextual representations that the decoder can attend to. The conditioning on $y_{<t}$ shows the autoregressive nature of the decoder: when predicting token $t$, the model can see all previous target tokens it has generated.

This is the standard sequence-to-sequence cross-entropy loss, where the encoder processes the corrupted input and the decoder generates the target autoregressively. The encoder-decoder architecture is important here: the encoder builds rich bidirectional representations of the corrupted input (processing both left and right context simultaneously), while the decoder generates the target one token at a time, attending to both the encoder representations and its own previous outputs.

The strength of this formulation lies in its simplicity. Despite the somewhat complex corruption procedure, the training objective is identical to any other sequence-to-sequence task like machine translation. The model learns to generate the correct output sequence given the input sequence, and the span corruption simply defines what those input-output pairs look like during pre-training. This means the same architecture and training infrastructure developed for seq2seq tasks applies directly to pre-training.

The T5 paper compared span corruption against several alternatives, including BERT-style token masking. Span corruption consistently performed better for several reasons, each stemming from basic differences in what the model must learn.

When BERT masks tokens, each masked position is predicted independently using bidirectional context. The model produces a probability distribution over the vocabulary for each masked position, but these predictions don't need to cohere with each other. If "The [MASK] [MASK] jumps" has two masked positions, BERT predicts each independently. It might assign high probability to "brown" for the first position and high probability to "cat" for the second, even though "brown cat" would be unusual in context while "brown fox" would be natural.

Span corruption eliminates this incoherence problem entirely. When the decoder generates "brown fox" to fill a single span, it produces "fox" after having already committed to "brown." The autoregressive generation ensures that multi-token predictions are internally consistent. The model learns which words fit each position and which word sequences form natural phrases.

The key advantages of span corruption include:

• Richer predictions: Generating multiple consecutive tokens requires modeling local coherence, not just fill-in-the-blank vocabulary matching
• Computational efficiency: Replacing spans with single sentinels shortens inputs, letting larger batch sizes or longer sequences
• Better calibration: The model learns to generate varying-length outputs, preparing it for downstream generation tasks

The corruption rate of 15% with mean span length 3 emerged from extensive testing as an effective balance. Higher corruption rates give more training signal per example but can make the task too difficult. Lower rates are easier but give less learning signal. The mean span length of 3 ensures a good mix of single-token, short-phrase, and occasional longer-phrase predictions, exposing the model to reconstruction challenges at multiple scales.

The efficiency benefit is particularly important. Consider a 512-token input with 15% corruption (approximately 77 tokens masked). With BERT-style masking, the input remains 512 tokens, and the model predicts 77 tokens. With T5-style span corruption, assuming approximately 26 spans (77 tokens ÷ 3 mean length), the corrupted input shrinks to roughly 461 tokens (512 - 77 + 26). The encoder processes fewer tokens while the decoder still predicts the same 77 tokens of content. This efficiency compounds across the billions of training examples in pre-training.