BART Pre-training › Visualizing Corruption Density

Michael BrenndoerferOctober 19, 20253 min read
Language AI HandbookAll chapters
Part 25 of 67Encoder-Decoder Models
Chapter 181BART Pre-training › Visualizing Corruption Density
Reading section 12 of 16 · Visualizing Corruption Density · 3 minSection 12 of 16 · 3 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 181 of 513

Section 12 of 16 · 3 min read

BART Pre-training › Visualizing Corruption DensityOne section · Sections in this chapter12 / 16 ↓
Sections in this chapter16
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Understanding how different corruption schemes distribute their modifications helps build intuition about what the model learns.

Out[27]:
Visualization
Histogram showing distribution of masked token fractions.
Token masking (p=0.15) affects ~15% of tokens.
Histogram showing distribution of deleted token fractions.
Token deletion (p=0.15) removes ~15% of tokens.
Histogram showing distribution of tokens in corrupted spans.
Text infilling (30%, λ=3) corrupts ~30% of tokens with higher variance.

Token masking and deletion show similar distributions centered around their target probability. Text infilling affects more tokens (30% target) and shows higher variance due to the stochastic span lengths.some samples have many short spans while others have fewer long spans.