T5 Pre-training › Visualizing Span Corruption Statistics

Michael BrenndoerferAugust 15, 20255 min read
Language AI HandbookAll chapters
Part 25 of 67Encoder-Decoder Models
Chapter 178T5 Pre-training › Visualizing Span Corruption Statistics
Reading section 8 of 11 · Visualizing Span Corruption Statistics · 5 minSection 8 of 11 · 5 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 178 of 513

Section 8 of 11 · 5 min read

T5 Pre-training › Visualizing Span Corruption StatisticsOne section · Sections in this chapter8 / 11 ↓
Sections in this chapter11
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let's examine the distribution of span lengths and corruption patterns:

Out[18]:
Visualization
Bar chart showing sampled span length frequencies from 1 to 15, overlaid with the theoretical geometric distribution for p equals one third.
Distribution of span lengths sampled from a geometric distribution with mean 3. Shorter spans are most common, with probability decreasing geometrically.

The empirical distribution closely matches the theoretical geometric distribution. About one-third of spans contain just a single token, making span corruption somewhat similar to token-level masking in many cases. However, the longer spans give useful training signal for learning longer-range dependencies.

Out[19]:
Visualization
Line chart showing corrupted input length decreasing from 512 toward shorter lengths as corruption rate increases from 5% to 50%, with a red dashed line at T5's default 15%.
Effect of corruption rate on input compression. Higher corruption rates lead to shorter corrupted inputs.
Line chart showing expected number of corrupted spans increasing as corruption rate rises from 5% to 50%, with a red dashed line at T5's default 15%.
Effect of corruption rate on number of spans. More spans are created as corruption increases.

At T5's default 15% corruption rate with mean span length 3, we expect about 25 spans in a 512-token sequence. The input length shrinks to roughly 460 tokens because each multi-token span is replaced by a single sentinel.

Out[20]:
Visualization
Combined bar-and-line chart comparing T5 and BERT-style masking across corruption rates from 10 to 30 percent: encoder input lengths appear as paired bars, target and prediction lengths as lines, and the original 512-token length as a horizontal reference.
Training efficiency comparison: tokens processed by encoder versus tokens predicted by decoder across different corruption rates. T5''s span corruption reaches favorable ratios.