Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Span Corruption › Span Corruption Variants

Michael BrenndoerferUpdated July 11, 20257 min read

Language AI HandbookAll chapters

Part 22 of 67Pre-training Objectives

Chapter 158Span Corruption › Span Corruption Variants

Reading section 8 of 14 · Span Corruption Variants · 7 minSection 8 of 14 · 7 min

Section 8 of 14 · 7 min read

Span Corruption › Span Corruption VariantsOne section · Sections in this chapter8 / 14 ↓

Sections in this chapter14

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Researchers have explored several variations of the basic span corruption approach. Each variant modifies one or more of the core design choices: the span length distribution, the corruption rate, or the attention pattern used during training. Understanding these variants illuminates the space of possible pretraining objectives and helps explain design choices in models beyond T5.

Uniform vs. Geometric Span LengthsLink Copied

While T5 uses geometric distribution, some work explores uniform distributions. With uniform sampling between 1 and $k$ , every span length up to $k$ is equally likely:

In[23]:

Code

def sample_span_lengths_uniform(num_spans, min_length=1, max_length=5):
    """Sample span lengths from uniform distribution."""
    return np.random.randint(min_length, max_length + 1, size=num_spans)


def sample_span_lengths_geometric(num_spans, mean_length=3):
    """Sample span lengths from geometric distribution."""
    p = 1 / mean_length
    return np.random.geometric(p, size=num_spans)

In[24]:

Code

n_samples = 5000

uniform_spans = sample_span_lengths_uniform(n_samples, max_length=5)
geometric_spans = sample_span_lengths_geometric(n_samples, mean_length=3)

Out[25]:

Visualization

Bar chart showing uniform distribution of span lengths from 1 to 5. — Uniform distribution gives equal probability to all span lengths in the range 1-5. Each length is sampled with equal frequency. This gives guaranteed exposure to longer spans but no emphasis on the shorter spans that dominate natural phrase structure.

Bar chart showing geometric distribution of span lengths with exponential decay. — Geometric distribution favors shorter spans with an exponential tail allowing occasional long ones. The rapidly decaying probabilities concentrate training effort on short and medium-length reconstructions while still exposing the model to longer spans.

Geometric distributions produce more short spans but allow occasional long ones. Uniform distributions guarantee exposure to longer spans but may waste capacity on trivially short reconstructions.

The practical implication is that the choice of distribution should match the downstream task distribution. If the primary downstream tasks involve short phrase completions and entity identification, the geometric distribution with a small mean aligns well. If the tasks involve longer document-level generation or summarization where multi-sentence spans matter, experimenting with higher mean values or distributions with heavier tails may be worthwhile.

We can also explore how different mean span lengths affect the distribution shape. Higher means produce flatter distributions with more emphasis on longer spans.

Out[26]:

Visualization

Three overlapping bar charts showing geometric distributions with means 2, 3, and 5, with lower means having steeper decay. — Effect of mean span length on the geometric distribution shape. Mean 2 concentrates nearly half of all spans at length 1. This gives mostly token-level signal. Mean 5 spreads probability substantially across longer spans, training the model on more complex multi-word reconstructions. T5's choice of mean 3 occupies the middle ground, balancing token-level and phrase-level training.

With $\mu = 2$ , nearly half of all spans are single tokens. This provides mostly token-level signal. With $\mu = 5$ , the distribution flattens clearly. This exposes the model to longer spans more frequently but at the cost of fewer distinct spans per sequence. T5's choice of $\mu = 3$ balances these extremes.

Corruption Rate VariationsLink Copied

The original T5 paper tested corruption rates from 10% to 50%. Higher rates create shorter inputs but longer targets:

In[27]:

Code

def analyze_corruption_rate(
    seq_length, corruption_rate, mean_span=3, num_trials=1000
):
    """Analyze the effect of corruption rate on sequence lengths."""
    input_lengths = []
    target_lengths = []

    for _ in range(num_trials):
        mask = select_corruption_mask(seq_length, corruption_rate, mean_span)
        spans = get_span_boundaries(mask)

        # Input length = original - corrupted + num_sentinels
        corrupted_count = mask.sum()
        input_len = seq_length - corrupted_count + len(spans)
        input_lengths.append(input_len)

        # Target length = corrupted + num_sentinels
        target_len = corrupted_count + len(spans)
        target_lengths.append(target_len)

    return np.mean(input_lengths), np.mean(target_lengths)

In[28]:

Code

seq_length = 512
corruption_rates = [0.1, 0.15, 0.2, 0.25, 0.3, 0.4, 0.5]

results = []
for rate in corruption_rates:
    input_len, target_len = analyze_corruption_rate(seq_length, rate)
    results.append(
        {
            "rate": rate,
            "input_length": input_len,
            "target_length": target_len,
            "total": input_len + target_len,
        }
    )

Out[29]:

Visualization

Line plot showing input length decreasing and target length increasing as corruption rate rises from 10% to 50%. — Effect of corruption rate on input and target sequence lengths for a 512-token sequence. As corruption increases, the encoder input shrinks (fewer tokens to process) while the target grows (more spans to reconstruct). The total combined length grows beyond the original as the overhead from sentinel tokens compounds.

At 15% corruption, the input shrinks modestly while the target remains manageable. Higher rates shift more content to the target, which can slow training due to longer autoregressive generation.

Prefix LM VariantLink Copied

Some models combine span corruption with prefix language modeling. The uncorrupted prefix receives bidirectional attention, while corrupted spans use causal attention:

Input: "The quick brown" + <extra_id_0> + "jumps over" + <extra_id_1>

Attention:

"The quick brown": Bidirectional (full visibility)
Sentinels and after: Causal (left-to-right only)

This hybrid approach lets the model use bidirectional context for understanding while maintaining generative capability. The intuition is that a strong language model needs both skills: reading and understanding text (served by bidirectional attention) and generating new text (served by causal autoregressive generation). The prefix LM variant approximates this by giving the prefix full attention while treating the generation phase causally.

UL2 (Unifying Language Learning Paradigms), introduced by Google in 2022, extended this idea further by mixing multiple denoising objectives during pretraining, including span corruption, prefix language modeling, and extreme long-span corruption. The UL2 paper showed that no single objective is universally best across all tasks and that a mixture of objectives produces more versatile models. This finding validated the span corruption design while also showing its limitations: a model pretrained only on span corruption will underperform on tasks that require strong causal generation from an open-ended prompt.