BART Pre-training › Limitations and Impact

BART's pre-training approach has limitations that shaped subsequent research.

The full-sequence reconstruction objective is computationally expensive. Unlike T5, which generates only corrupted spans, BART's decoder processes every token of the original sequence. For long documents, this increases training cost significantly. The autoregressive decoder also prevents parallelization during training, unlike the encoder which processes all positions simultaneously.

The fixed corruption scheme may not be optimal for all downstream tasks. BART's text infilling with sentence permutation works well for summarization and generation, but other corruptions might better suit specific applications. More recent work has explored task-adaptive pre-training.

BART's bidirectional encoder limits its effectiveness for pure language modeling tasks. The model cannot directly perform next-token prediction like GPT because its encoder expects to see corrupted versions of complete sequences, not prefixes of text. This makes BART less suitable for applications requiring left-to-right generation from prompts without conditioning input.

Despite these limitations, BART demonstrated that encoder-decoder models could match or exceed BERT and GPT on their respective strengths by using appropriate pre-training corruptions. This finding influenced subsequent work on unified models that handle diverse tasks within a single architecture.

The span-based corruption approach has proven durable. Modern models continue to use variants of span corruption, with innovations focusing on how to select span boundaries, what to replace spans with, and how to balance local and global corruptions. The core idea remains central to pre-training design.