Denoising Objectives › Limitations and Impact

Denoising objectives have transformed language model pretraining, but they come with tradeoffs that shape their practical applications.

The most fundamental limitation is the mismatch between pretraining and downstream tasks. During pretraining, the model always receives corrupted input and must produce the original. During fine-tuning and inference, the model receives clean input and must produce novel output. This gap means that denoising models may not immediately transfer their reconstruction skills to generation tasks without fine-tuning. BART addresses this partially through its autoregressive decoder, which learns generation patterns even during denoising, but the task distribution shift remains a concern for zero-shot applications. A model that has spent all of its pretraining learning to reconstruct from noise may not have developed the same degree of open-ended generative fluency as a model trained purely on next-token prediction over clean text.

Computational efficiency presents another tradeoff. Aggressive corruption reduces the input sequence length (especially with text infilling), which speeds up encoding. However, the decoder must still generate the full original sequence, and the encoder-decoder architecture is more expensive than encoder-only or decoder-only models of equivalent capacity. For pure understanding tasks, BERT-style encoders may be more efficient. For pure generation tasks, GPT-style decoders may be preferable. Denoising models like BART occupy a middle ground that excels when both capabilities are needed, such as in conditional text generation where the model must understand an input document and produce a coherent output based on it.

Key practical considerations include:

• Corruption choice matters: Different corruptions suit different downstream tasks. Text infilling helps summarization but may not help classification. Empirical tuning is often necessary.

• Sentence boundaries required: Sentence permutation requires reliable sentence segmentation. For domains with unusual punctuation or formatting, this corruption may introduce artifacts.

• Length changes complicate batching: Token deletion and infilling change sequence lengths, requiring dynamic padding or variable-length batching infrastructure.

• Hyperparameter sensitivity: The interaction between mask probability, mean span length, and sentence permutation means that the optimal corruption configuration varies by domain and downstream task. What works for news summarization may not be optimal for code generation or scientific question answering.

Despite these limitations, denoising objectives work well in practice. BART achieved state-of-the-art results on summarization and translation, as well as question answering, when released. The insight that diverse corruptions produce versatile models has influenced subsequent work like T5, mBART, and PEGASUS. Models trained with denoising objectives often transfer better to generation tasks than MLM-only models, while maintaining competitive understanding performance. The PEGASUS model, for instance, adapted the infilling idea specifically for summarization by masking entire sentences that were deemed important (using a sentence-level importance score), training the model to generate the most salient content from the remaining context. This shows how the general principle of denoising can be adapted and specialized for particular applications beyond what BART originally demonstrated.