BART Pre-training › Combining Corruption Schemes
Language AI HandbookAll chapters BART Pre-training › Combining Corruption SchemesOne section · Sections in this chapter8 / 16 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
BART's key experimental contribution was systematically testing different combinations of these corruption schemes. Not all combinations are equally effective, and some corruptions complement each other while others are redundant. Understanding these interactions helps explain BART's final design choices and provides guidance for future model development.
Experimental Findings
The BART paper evaluated combinations across several downstream tasks. Key findings include:
Text infilling dominates. When used alone, text infilling matched or exceeded other single-scheme corruptions across most tasks. The combination of length prediction and span reconstruction provides strong learning signals that transfer effectively to generation tasks.
Sentence permutation adds modest gains. Combining text infilling with sentence permutation slightly improved results on some tasks, particularly those requiring document understanding. However, the gains were small compared to infilling alone.
Token deletion is redundant with infilling. Text infilling subsumes token deletion because zero-length spans in infilling are equivalent to deletions. Combining them provides little benefit.
Document rotation hurts performance. Adding document rotation to other schemes generally degraded results. The task may be too difficult or too different from downstream applications.
Token masking alone is inferior. Pure masking performs worse than infilling because it cannot teach length prediction. Each mask always corresponds to exactly one token.
The Final BART Recipe
Based on these experiments, the standard BART pre-training uses:
- Text infilling with and approximately 30% of tokens in corrupted spans
- Sentence permutation applied to all documents
This combination balances local token prediction with global structure learning while avoiding the problems with rotation and the redundancy of separate deletion.
def bart_corruption(
text: str,
mask_ratio: float = 0.30,
poisson_lambda: float = 3.0,
sentence_end_pattern: str = r"(?<=[.!?])\s+",
mask_token: str = "[MASK]",
) -> Tuple[str, str]:
"""Apply BART's combined corruption scheme.
Combines text infilling and sentence permutation.
Returns:
Tuple of (corrupted_text, original_text)
"""
# Step 1: Sentence permutation
sentences = re.split(sentence_end_pattern, text.strip())
sentences = [s.strip() for s in sentences if s.strip()]
if len(sentences) > 1:
random.shuffle(sentences)
permuted_text = " ".join(sentences)
# Step 2: Text infilling on permuted text
tokens = permuted_text.split()
corrupted_tokens, _ = text_infilling(
tokens,
mask_ratio=mask_ratio,
poisson_lambda=poisson_lambda,
mask_token=mask_token,
)
corrupted_text = " ".join(corrupted_tokens)
return corrupted_text, textnp.random.seed(42)
random.seed(42)
document = """Natural language processing enables computers to understand human text.
Deep learning has driven recent advances in the field.
BART combines bidirectional and autoregressive training."""
corrupted, original = bart_corruption(document, mask_ratio=0.25)Original: Natural language processing enables computers to understand human text. Deep learning has driven recent advances in the field. BART combines bidirectional and autoregressive training. Corrupted (permuted + infilled): [MASK] recent advances in the [MASK] processing enables computers to understand human text. BART combines bidirectional and autoregressive [MASK]
The combined corruption produces text that is both structurally shuffled and locally corrupted. The decoder faces the full challenge of restoring both sentence order and masked content.