Span Corruption › T5-Style Training

T5 uses span corruption within an encoder-decoder framework. The corrupted input feeds into the encoder, while the decoder autoregressively generates the target sequence. This setup naturally handles variable-length span reconstruction.

The encoder processes the corrupted input sequence, producing hidden representations that capture the context around each sentinel:

$$H_{\text{enc}} = \text{Encoder}(x_1, \ldots, \text{<extra\_id\_0>}, \ldots, x_n)$$

where:

• $H_{\text{enc}}$: the encoder's output hidden states, a matrix of shape (sequence length $\times$ hidden dimension)
• $x_1, \ldots, x_n$: the corrupted input tokens, including uncorrupted tokens and sentinel placeholders
• $\text{<extra\_id\_0>}$: a sentinel token marking where a span was removed

The encoder uses full bidirectional self-attention: every token can attend to every other token. This means the representation at the <extra_id_0> position has access to all the surrounding context, both the tokens that appear before the sentinel and those that appear after. This bidirectional view is a significant advantage over causal decoder-only models, which can only see leftward context. When predicting what phrase fills a gap, the context to the right of the gap is often just as informative as the context to the left, and the encoder captures both.

The decoder then generates the target sequence autoregressively, conditioned on the encoder's hidden states:

$$P(y_t \mid y_{<t}, H_{\text{enc}}) = \text{Decoder}(y_{<t}, H_{\text{enc}})$$

where:

• $y_t$: the token being predicted at position $t$ in the target sequence
• $y_{<t}$: all previously generated target tokens $(y_1, y_2, \ldots, y_{t-1})$
• $P(y_t \mid y_{<t}, H_{\text{enc}})$: the probability distribution over the vocabulary for the next token

During training, we use teacher forcing: the decoder receives the true previous tokens rather than its own predictions. Teacher forcing is important for span corruption because the target sequence can be quite long (dozens of tokens across multiple spans), and sampling from the model's own distributions during training would introduce compounding errors that would destabilize learning. By feeding the ground-truth tokens at each decoder step, the model receives clean, unambiguous gradient signal throughout the target sequence.

The training objective is standard cross-entropy loss over the target sequence:

$$\mathcal{L} = -\sum_{t=1}^{T} \log P(y_t \mid y_{<t}, H_{\text{enc}})$$

where:

• $\mathcal{L}$: the total loss for this training example
• $T$: the length of the target sequence
• $\log P(y_t \mid \cdot)$: the log probability assigned to the correct token at each position

This loss encourages the model to assign high probability to the actual tokens that were corrupted, learning to reconstruct spans from context. Because the target includes both sentinel tokens and the actual span content, the model learns two things simultaneously: when to begin generating a new span (at sentinel boundaries) and what content to generate within each span.

Span corruption works well with encoder-decoder models for several interconnected reasons.

Bidirectional encoding is perhaps the most important. The encoder sees all uncorrupted context simultaneously, enabling rich representations of what surrounds each sentinel. Unlike a causal language model that can only look left, the encoder integrates both left and right context at every position, producing representations that carry far more information about the missing span. For named entity recognition, for instance, the context to the right of a person's name can be just as disambiguating as the context to the left.

Autoregressive decoding complements this by enabling the model to generate spans of variable length coherently. The decoder generates spans token by token, learning proper phrase structure and coherence within each span. Because each generated token conditions on all previous tokens in the target, the decoder naturally models intra-span dependencies: the second word of a phrase is influenced by the first, the third by both, and so on.

Natural length handling emerges from this architecture without any special design. Spans of different lengths produce targets of different lengths, which encoder-decoder models handle gracefully through their sequence-to-sequence framing. A decoder-only model using span corruption as a fill-in-the-blank task would need to know in advance how many tokens to generate for each gap, which is not available during inference.

Decoder-only models can also use span corruption, treating the task as infilling. The corrupted input and target concatenate with appropriate separators, and the model learns to continue after each sentinel. Some newer models like FIM (Fill in the Middle) explore exactly this setting. The trade-off is that decoder-only models must still predict a fixed-length span or use special boundary tokens, whereas the encoder-decoder design handles this naturally through its two-part architecture.

Let's compare the training signals from span corruption versus token-level MLM: