Denoising Objectives › Comparing Corruption Strategies
Michael BrenndoerferUpdated July 14, 20253 min read
Language AI HandbookAll chapters Part 22 of 67Pre-training Objectives
Denoising Objectives › Comparing Corruption StrategiesOne section · Sections in this chapter10 / 15 ↓
Sections in this chapter15
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Different corruptions teach different aspects of language understanding. Let's compare how each strategy transforms the same input.
In[32]:
Codedef compare_corruptions(text, seed=42):
"""Compare all corruption strategies on the same text."""
tokens = text.split()
results = {}
# Token deletion
np.random.seed(seed)
deleted, _ = apply_token_deletion(tokens, deletion_prob=0.2)
results["Token Deletion"] = " ".join(deleted)
# Token shuffling
np.random.seed(seed)
shuffled, _ = apply_token_shuffling(tokens, shuffle_distance=3)
results["Token Shuffling"] = " ".join(shuffled)
# Text infilling
np.random.seed(seed)
infilled, _, _ = apply_text_infilling(
tokens, mask_prob=0.25, mean_span_length=2
)
results["Text Infilling"] = " ".join(infilled)
# Sentence permutation
np.random.seed(seed)
permuted, _ = apply_sentence_permutation(text)
results["Sentence Permutation"] = permuted
return resultsIn[33]:
Codesample = "The quick brown fox jumps. It leaps over the lazy dog. The dog wakes up surprised."
comparisons = compare_corruptions(sample)Out[34]:
ConsoleOriginal: The quick brown fox jumps. It leaps over the lazy dog. The dog wakes up surprised. ============================================================ Token Deletion: The quick brown fox over the lazy The dog wakes Token Shuffling: The quick brown jumps. fox It leaps over the dog. lazy wakes The dog up surprised. Text Infilling: The quick brown fox jumps. It leaps [MASK] [MASK] The dog wakes up surprised. Sentence Permutation: The quick brown fox jumps. It leaps over the lazy dog. The dog wakes up surprised.
Each corruption breaks different properties:
- Token deletion: Removes specific words, breaking local completeness
- Token shuffling: Scrambles word order, breaking syntax
- Text infilling: Hides contiguous chunks, breaking both content and length
- Sentence permutation: Reorders sentences, breaking discourse flow
The choice of corruption depends on the target application. For generation tasks, text infilling works best because it trains the decoder to produce variable-length outputs. For understanding tasks, combining multiple corruptions teaches the model to recover several kinds of structure.
Out[35]:
Visualization