Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Span Corruption › Computational Benefits

Michael BrenndoerferUpdated July 11, 20255 min read

Language AI HandbookAll chapters

Part 22 of 67Pre-training Objectives

Chapter 158Span Corruption › Computational Benefits

Reading section 9 of 14 · Computational Benefits · 5 minSection 9 of 14 · 5 min

Section 9 of 14 · 5 min read

Span Corruption › Computational BenefitsOne section · Sections in this chapter9 / 14 ↓

Sections in this chapter14

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Span corruption offers surprising computational advantages over token-level masking. These benefits arise not from architectural changes but purely from the way the corruption objective reshapes the sequences that the model must process.

Shorter SequencesLink Copied

Because multiple tokens collapse into single sentinels, the encoder processes fewer tokens. For a 512-token sequence with 15% corruption and mean span length 3:

Original tokens corrupted: ~77 tokens
Sentinels added: ~26 tokens
Net input length: ~461 tokens (10% reduction)

This reduction compounds across the quadratic attention mechanism. Attention cost scales as $O(n^2)$ , where $n$ is the sequence length. A 10% length reduction (from $n$ to $0.9n$ ) yields roughly 19% savings in attention computation, since $(0.9)^2 = 0.81$ .

In practice, at the scale T5 operates, this matters. T5-11B was pretrained on approximately one trillion tokens. A 10% reduction in per-example computation translates directly to fewer GPU-hours needed to process the same amount of text, or equivalently, the ability to process 10% more text within the same compute budget. At billion-dollar pretraining costs, even single-digit percentage efficiency improvements are meaningful.

Shorter TargetsLink Copied

The target sequence contains only corrupted spans plus sentinels. With 15% corruption:

Target length: ~77 (spans) + ~26 (sentinels) = ~103 tokens

The decoder processes 103 tokens instead of 512, dramatically reducing generation cost during training. Autoregressive generation in the decoder is sequential and cannot be parallelized across positions (each token depends on all previous tokens). Reducing the target from 512 to 103 tokens makes the sequential decoding step roughly 5x faster, which is a much more significant speedup than the encoder efficiency gain.

The combination of a shorter encoder input and a dramatically shorter decoder target means span corruption's computational profile looks quite different from full-sequence causal language modeling. While a causal LM must process the entire 512-token sequence sequentially in the decoder, span corruption processes 461 tokens in parallel in the encoder and 103 tokens sequentially in the decoder. This asymmetric split is computationally favorable because parallel operations are so much cheaper on modern hardware than sequential ones.

Training Efficiency ComparisonLink Copied

Let's quantify the computational savings:

In[30]:

Code

def compute_training_cost(seq_length, corruption_rate, mean_span):
    """
    Estimate relative computational cost for different pretraining approaches.
    Uses simplified model where cost ~ sequence_length^2 for attention.
    """
    # Calculate expected lengths
    num_corrupted = int(seq_length * corruption_rate)
    num_spans = max(1, num_corrupted // mean_span)

    # Span corruption
    span_input_len = seq_length - num_corrupted + num_spans
    span_target_len = num_corrupted + num_spans
    span_cost = span_input_len**2 + span_target_len**2  # Encoder + Decoder

    # Standard MLM (encoder only, full sequence)
    mlm_cost = seq_length**2

    # Causal LM (decoder only, full sequence)
    clm_cost = (
        seq_length**2
    )  # Simplified; actual is O(n^2/2) due to causal mask

    return {
        "span_corruption": span_cost,
        "mlm": mlm_cost,
        "causal_lm": clm_cost,
        "span_input_len": span_input_len,
        "span_target_len": span_target_len,
    }

In[31]:

Code

costs = compute_training_cost(512, 0.15, 3)

Out[32]:

Console

Sequence length: 512
Corruption rate: 15%, Mean span: 3

Span corruption:
  Input length:  461
  Target length: 101
  Relative cost: 84.96% of MLM

MLM/Causal LM cost: 100% (baseline)

Let's visualize these computational trade-offs across different corruption rates to understand when span corruption provides the greatest efficiency gains.

In[33]:

Code

# Compute costs across corruption rates
cost_comparison = []
for rate in [0.1, 0.15, 0.2, 0.25, 0.3]:
    costs = compute_training_cost(512, rate, 3)
    cost_comparison.append(
        {
            "rate": rate,
            "relative_cost": costs["span_corruption"] / costs["mlm"] * 100,
            "input_len": costs["span_input_len"],
            "target_len": costs["span_target_len"],
        }
    )

Out[34]:

Visualization

Bar chart showing relative computational cost at different corruption rates, all below the 100% MLM baseline. — Relative computational cost of span corruption compared to MLM baseline (100%) at different corruption rates. All corruption rates achieve meaningful savings because the encoder processes a compressed input and the decoder handles only the small target span sequence. The U-shaped trend arises from competing effects: higher corruption shortens the input more but lengthens the target.

Span corruption achieves meaningful computational savings while still exposing the model to diverse reconstruction challenges. At the standard 15% corruption rate, we save roughly 15% of computation compared to MLM. The encoder sees nearly the full context (minus corrupted tokens), while the decoder focuses only on what needs reconstruction.