BART Pre-training › The Denoising Framework

BART's pre-training objective is simple: given a corrupted document $\tilde{x}$, reconstruct the original document $x$. This turns the abstract problem of "learning language" into a concrete task: undo the corruption and recover the original text. The approach works because the objective creates rich learning signals based on how we corrupt the input.

The model minimizes negative log-likelihood loss:

where:

• $\mathcal{L}$: the loss function to minimize (negative log-likelihood)
• $x$: the original uncorrupted document
• $|x|$: the number of tokens in the original document
• $x_t$: the $t$-th token of the original document
• $x_{<t}$: all tokens before position $t$ (the decoder's previous outputs)
• $\tilde{x}$: the corrupted version of the document (encoder input)
• $P(x_t \mid x_{<t}, \tilde{x})$: the probability the model assigns to the correct token $x_t$

Consider what happens at each decoding step. The model uses two information sources: the corrupted input $\tilde{x}$ processed by the encoder, and the previously generated tokens $x_{<t}$. With these, the model predicts the probability distribution over all possible next tokens. We measure how much probability mass it assigns to the correct token $x_t$. The logarithm converts multiplicative probabilities into additive quantities. The negative sign changes our goal from maximizing likelihood to minimizing loss, which is standard in optimization.

This is cross-entropy loss over the original tokens, conditioned on both the previously generated tokens and the corrupted input. The encoder processes $\tilde{x}$, and the decoder generates $x$ token by token. The summation over all $|x|$ tokens means we get a training signal at every position in the sequence, not just corrupted ones. This dense supervision distinguishes BART from BERT, which only computes loss at masked positions.

The key design choice is selecting the noising function $g$ that turns $x$ into $\tilde{x} = g(x)$. This function determines what the model learns during pre-training. Different noising functions create different learning signals. Some emphasize local word relationships, others stress document structure, and some require the model to infer the length of missing content. If corruption is too easy to reverse, the model learns little because it can trivially recover the original without developing strong linguistic understanding. If corruption is too difficult or destroys too much structure, the model cannot learn meaningful patterns because there's insufficient signal remaining to guide reconstruction. BART's innovation was systematically exploring this design space to identify corruptions that hit the sweet spot: challenging enough to force learning but structured enough to permit it.

Unlike BERT, which only predicts masked tokens, BART's decoder must generate the entire original sequence. This architectural choice affects training. The model receives a training signal for every token position, not just corrupted ones. Even "easy" positions where the input was not corrupted contribute to learning. The decoder must copy these tokens and position them correctly within the output sequence. Unlike T5, which uses a specific span corruption scheme with sentinel tokens, BART explores a broader range of corruption strategies and reconstructs exact original text rather than just the corrupted spans. This full-sequence reconstruction teaches the model to blend generated content with copied content, a skill relevant to tasks like summarization where the output combines novel phrases with material drawn from the source.