Denoising Objectives › Denoising Objectives

Language models learn by predicting missing or corrupted text. Masked language modeling replaces tokens with [MASK], and span corruption hides contiguous chunks. But what if we corrupted text in more diverse ways? Token deletion, shuffling, sentence permutation, and document rotation all introduce different types of noise that force models to learn different aspects of language structure.

Denoising objectives generalize the idea of text reconstruction. Instead of a single corruption strategy, they apply multiple transformations that break different properties of natural text. To recover the original, the model must learn word order, sentence boundaries, document structure, and semantic coherence. BART (Bidirectional and Auto-Regressive Transformers) pioneered this approach, showing that combining diverse noise types produces models that excel at both understanding and generation.

The core intuition behind this design is that different corruption types create different learning pressures. When you randomly mask a single token, you train the model to use local context to fill in a blank. But when you delete a token entirely, the model faces a harder challenge: it must figure out what word belongs there and where the gap is at all. When you permute sentences, you force the model to reason about causality, temporal sequence, and rhetorical flow at a level entirely beyond individual words. Each corruption type is essentially a different exam question testing a different facet of language understanding.

Think of it this way: a student who only ever practices fill-in-the-blank exercises will develop a narrow skill. A student who also practices reordering scrambled sentences, reconstructing documents from shuffled paragraphs, and expanding abbreviated summaries develops a much richer and more flexible understanding of how language works. The same principle applies to language model pretraining. By exposing the model to a diverse curriculum of reconstruction challenges, denoising objectives build representations that generalize across a wide range of downstream tasks.

In this chapter, we'll explore the major denoising transformations, understand what each one teaches the model, implement them from scratch, and see how combining them creates versatile language models.