Denoising Objectives › Summary

Denoising objectives train models to reconstruct original text from corrupted inputs. By applying diverse corruption strategies, these objectives expose models to a rich curriculum of reconstruction challenges that build multi-level language understanding, from individual word choice to discourse-level coherence. This chapter covered the major corruption strategies and their effects:

• Token deletion removes tokens entirely, forcing the model to identify missing content and insert it at the correct positions. Unlike masking, deletion also requires inferring the locations of gaps, making it a harder and more position-sensitive task.

• Token shuffling permutes word order within a local window, teaching the model syntax and word order constraints. The shuffle distance controls whether the model focuses on local phrase structure or longer-range sentence organization.

• Sentence permutation reorders sentences within documents, developing understanding of discourse structure and coherence. This corruption is uniquely effective for tasks that require reasoning about causality, temporal order, and rhetorical organization across multiple sentences.

• Document rotation moves content from the beginning to the end, training sensitivity to document boundaries and openings. While showing smaller benefits than other corruptions in BART's ablation studies, it is particularly useful for genre-aware tasks.

• Text infilling replaces spans with single mask tokens, combining content prediction with length prediction for flexible generation. The geometric distribution of span lengths ensures a mix of token-level and phrase-level reconstruction challenges.

• BART-style combination uses text infilling plus sentence permutation to train models that excel at both understanding and generation. This combination covers both local and global structure, producing representations that transfer effectively to summarization and translation, as well as question answering.

The choice of corruption strategy depends on the target application. Text infilling suits generation tasks, while combining multiple corruptions produces general-purpose models that recover several kinds of structure. The encoder-decoder architecture of denoising models naturally supports both bidirectional encoding for understanding and autoregressive decoding for generation.

The next part of the book explores BERT and its variants, examining how encoder-only architectures apply masked language modeling to build powerful understanding models.