BART Pre-training › Summary
Language AI HandbookAll chapters BART Pre-training › SummaryOne section · Sections in this chapter15 / 16 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
BART's pre-training strategy frames language model training as denoising autoencoding. By systematically exploring five corruption schemes, the researchers identified effective combinations for learning both local token patterns and global document structure.
Text infilling proved to be the most important individual corruption, teaching the model to predict how many tokens each mask represents and to generate coherent multi-token spans. Sentence permutation adds modest gains for tasks requiring document-level understanding. The final BART recipe combines these two approaches while avoiding less effective corruptions.
The key design choices that distinguish BART from related approaches are:
- Full-sequence reconstruction: Unlike T5, BART generates the complete original text, giving training signal for every position
- Variable-length spans: Unlike BERT's one-to-one masking, BART's infilling teaches length prediction through many-to-one corruption.
- Combined structural and local corruption: Sentence permutation addresses discourse while infilling addresses token-level prediction
These pre-training choices directly enable BART's strong performance on generation tasks such as summarization and translation, as well as dialogue generation.