Denoising Objectives › Worked Example: Tracing a Single Document Through BART Corruption

To make the full pipeline concrete, let's trace a single document step by step through BART's combined corruption strategy and understand exactly what the model must learn at each stage.

Consider the following short document: "The sun rose over the mountains. The hikers began their ascent. By noon they reached the summit. The view was breathtaking."

This four-sentence document has a clear narrative structure: setting, action, culmination, reaction. The sentences flow logically from one to the next, with temporal markers ("By noon") and causal implications ("the view was breathtaking" follows "they reached the summit") binding them together.

When we apply sentence permutation, this document might become: "The view was breathtaking. The sun rose over the mountains. By noon they reached the summit. The hikers began their ascent." Now the reaction appears first, before the setting has been established. The temporal marker "By noon" loses its meaning because we haven't been told what morning activity the noon event follows. The document still contains all four sentences, but it no longer makes sequential sense.

Next, text infilling replaces 30% of the tokens with [MASK] placeholders using span lengths drawn from a geometric distribution. The permuted document might become: "The [MASK] breathtaking. The sun rose [MASK] mountains. By noon they [MASK] summit. The hikers [MASK] their ascent." Now the model faces a doubly corrupted input: the sentences are out of order, and each sentence has gaps of unknown length.

To reconstruct the original, the model must simultaneously: (1) recognize that the sentences are in the wrong order and reorder them as part of generating the target sequence, and (2) fill in the masked spans with the correct words and the correct number of words. The decoder must output "The sun rose over the mountains. The hikers began their ascent. By noon they reached the summit. The view was breathtaking." from the scrambled, gapped input.

This is an extraordinarily demanding task that requires the model to reason at multiple levels at once. Every training step on a document like this exercises the model's understanding of temporal sequence, discourse structure, argument structure within individual sentences, and the relationship between context and missing words. After seeing billions of such examples, the model develops the rich multi-level representations that make BART-style models effective across so many downstream tasks.