Span Corruption › Span Selection Strategies

Now that we understand what span corruption does conceptually, we need to answer a practical question: how do we decide which parts of a sequence to corrupt? This leads us to two interconnected design choices that significantly affect what the model learns.

Think of span corruption as a controlled demolition. We want to remove enough of the building (the text) to create a meaningful reconstruction challenge, but not so much that the remaining structure provides no clues about what was removed. This balance requires careful decisions about two quantities: how much total material to remove (the corruption rate) and how to divide that material into chunks (the span length distribution).

The two parameters interact. If we fix the corruption rate at 15% and increase the average span length, we need fewer, longer spans. The model encounters fewer reconstruction tasks per sequence but each one is harder. If we reduce the mean span length, we get more, shorter spans. The model encounters more reconstruction tasks but each one is easier and more local. The right balance depends on what downstream capabilities matter most.

The corruption rate $r$ specifies what fraction of the original tokens should end up inside corrupted spans. T5 adopts $r = 0.15$, matching BERT's masking rate, but the mechanics differ substantially from token-level masking.

In standard MLM, a 15% corruption rate means we independently flip a coin for each token, with 15% probability of masking. The result: exactly 15% of tokens become [MASK], scattered throughout the sequence. With span corruption, we instead select contiguous regions totaling approximately 15% of the sequence. The key difference is that these tokens are grouped, not isolated.

This grouping creates an interesting constraint. If we want to corrupt a fixed fraction of tokens, the number of spans we create depends on how long each span is. Suppose we have a sequence of $n$ tokens and want to corrupt fraction $r$ of them. If each span contains an average of $\mu$ tokens, simple arithmetic tells us how many spans we need:

$$\text{number of spans} \approx \frac{r \cdot n}{\mu}$$

where:

• $n$: the total number of tokens in the input sequence
• $r$: the corruption rate (fraction of tokens to corrupt, e.g., 0.15 for 15%)
• $\mu$: the average span length (e.g., 3 tokens per span)

The reasoning is direct: we need to "cover" $r \cdot n$ tokens with our spans. If each span covers $\mu$ tokens on average, we need $(r \cdot n) / \mu$ spans to achieve the target coverage.

Let's make this concrete. For a 512-token sequence with $r = 0.15$ and $\mu = 3$:

1. Tokens to corrupt: $0.15 \times 512 \approx 77$ tokens
2. Spans needed: $77 / 3 \approx 26$ spans
3. Input length after corruption: Each span gets replaced by one sentinel token, so the corrupted input contains $512 - 77 + 26 = 461$ tokens

This calculation reveals an important property: span corruption naturally compresses the input sequence. We remove 77 tokens but add only 26 sentinels, achieving a net reduction of 51 tokens (about 10% shorter). This compression has computational benefits we'll explore later.

The T5 paper explored corruption rates between 10% and 50%. The team found that performance changed little in the 10%-25% range, with 15% consistently performing well across tasks. Rates above 30% begin to remove too much context, making reconstruction harder but also noisier as a training signal. Very high rates (40%-50%) can hurt downstream performance because the model spends most of pretraining on a task so hard that it cannot receive a clear learning signal from the context.

The corruption rate tells us how much to remove; the span length distribution tells us how to partition that removal into individual spans. This choice significantly affects what patterns the model learns.

Consider the extremes. If all spans had length 1, span corruption would collapse to token-level masking: we'd have 77 isolated masks, each predicting a single token independently. If all spans had length 77, we'd have one massive gap containing over half the sequence. This provides almost no useful training signal.

The sweet spot lies somewhere between. We want a mix of span lengths that exposes the model to diverse reconstruction challenges: some single-token predictions (maintaining fine-grained language modeling ability), some short phrases (learning local coherence), and occasional longer spans (forcing the model to reason about larger structures).

T5 uses a geometric distribution to achieve this mix. The geometric distribution is the discrete analog of the exponential distribution, modeling a natural process: imagine flipping a biased coin at each position within a span, continuing until we get "heads" (stop) instead of "tails" (continue). The span length equals how many positions we visited before stopping.

Mathematically, if the probability of stopping at each step is $p$, the probability of a span having exactly $\ell$ tokens is:

$$P(\ell) = (1 - p)^{\ell - 1} \cdot p$$

where:

• $P(\ell)$: the probability of sampling a span of exactly $\ell$ tokens
• $\ell$: the span length (a positive integer: 1, 2, 3, ...)
• $p$: the stopping probability at each step, controlling the distribution's shape
• $(1 - p)^{\ell - 1}$: the probability of "continuing" for $\ell - 1$ steps before finally "stopping"

This formula captures a simple generative process: to get a span of length $\ell$, we need $\ell - 1$ consecutive "continue" decisions (each with probability $1 - p$) followed by one "stop" decision (with probability $p$).

The geometric distribution has a convenient property: its mean equals $1/p$. This gives us a direct way to control the average span length. If we want spans to average $\mu$ tokens, we simply set:

$$p = \frac{1}{\mu}$$

For T5's choice of $\mu = 3$, this gives $p = 1/3$. At each step within a span, there's a 1-in-3 chance of stopping. The resulting distribution strongly favors shorter spans while maintaining a "heavy tail" of longer ones:

• $P(\ell = 1) = 0.33$ (one-third of spans are single tokens)
• $P(\ell = 2) = 0.22$ (about one-fifth are two tokens)
• $P(\ell = 3) = 0.15$ (around one-sixth are three tokens)
• $P(\ell = 4) = 0.10$ (one-tenth are four tokens)
• $P(\ell \geq 5) = 0.20$ (about one-fifth contain five or more tokens)

This distribution creates a varied training curriculum. The model frequently encounters single-token predictions, maintaining its vocabulary knowledge. It regularly sees two and three-token spans, learning common phrases and local syntax. And it occasionally faces longer spans requiring compositional reasoning.

Why is the geometric distribution a natural choice here, rather than some other distribution with a similar mean? The geometric distribution has the "memoryless" property: the probability of stopping at the next step does not depend on how far we have already traveled within a span. This means there is no "target length" that spans tend toward. Each step within a span is a fresh, independent decision, which mirrors the structure of natural language to some extent: clauses do not have a predetermined length, and there is no linguistic pressure to end a phrase at exactly step three versus step four. The memoryless property keeps the sampling process simple and avoids introducing artificial length preferences.

Let's visualize this distribution by sampling 10,000 span lengths and plotting the empirical frequencies. This will confirm our theoretical calculations and reveal the characteristic exponential decay.