BART Pre-training › Comparison with Other Pre-training Approaches
Language AI HandbookAll chapters BART Pre-training › Comparison with Other Pre-training ApproachesOne section · Sections in this chapter9 / 16 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Understanding BART's pre-training requires comparing it to related approaches we've covered in earlier chapters.
BART vs. BERT
BERT's masked language modeling (covered in Part XXIV) masks 15% of tokens and predicts only those positions. Key differences:
| Aspect | BERT | BART |
|---|---|---|
| Architecture | Encoder-only | Encoder-decoder |
| Output | Masked positions only | Full sequence |
| Mask-to-token ratio | 1:1 | Variable (many:many) |
| Position signals | Mask tokens mark positions | No explicit position markers |
| Generation capability | Limited | Native |
BERT performs well at tasks requiring bidirectional understanding, such as classification and extraction, but struggles with generation because it was never trained to produce full sequences. BART maintains BERT's bidirectional encoder benefits while adding generation capability.
BART vs. T5
T5's span corruption (covered in Part XXV, Chapter 2) shares BART's span-based approach but differs in key ways:
| Aspect | T5 | BART |
|---|---|---|
| Corruption output | Replace spans with sentinels | Replace spans with single mask |
| Decoder output | Only corrupted spans | Full original sequence |
| Span identification | Sentinels mark boundaries | No explicit boundaries |
| Training signal | Only corrupted tokens | All tokens |
T5's sentinel-based approach is more efficient, but BART's full-sequence reconstruction provides better signal for tasks requiring exact output control, such as grammatical error correction or style transfer.
BART vs. GPT
GPT's causal language modeling (Part XXVIII) trains the model to predict the next token given all previous tokens. Comparing to BART:
| Aspect | GPT | BART |
|---|---|---|
| Input corruption | None (left context only) | Various noising schemes |
| Attention | Causal (unidirectional) | Encoder bidirectional, decoder causal |
| Learning signal | Next token prediction | Full reconstruction |
| Zero-shot capability | Strong | Moderate |
GPT's unidirectional approach prevents it from using future context during encoding. BART's encoder sees the full corrupted input bidirectionally, which helps for tasks where understanding the complete input matters.
