Denoising Objectives › The Denoising Autoencoder Framework

Denoising objectives treat pretraining as an autoencoding problem. The model receives corrupted input and must reconstruct the original. This differs from standard autoencoders, which simply copy input to output. By corrupting the input, we force the model to learn meaningful representations rather than trivial identity mappings.

The key insight is that the corruption function acts as a form of regularization. If the model could see the clean input directly, it could achieve zero loss by copying the input to the output token by token, learning nothing useful in the process. By hiding part of the information, corruption forces the model to build internal representations that encode the meaning and structure of the text, because those representations are what allow it to fill in what is missing. This is analogous to how studying by closing your notes and trying to reconstruct key ideas is more effective than simply re-reading them.

Formally, given original text $x$, we apply a corruption function $c(\cdot)$ to obtain noisy input $\tilde{x} = c(x)$. The model learns parameters $\theta$ to maximize the probability of recovering $x$ from $\tilde{x}$. The training objective minimizes the negative log-likelihood of reconstructing the original:

where:

• $\mathcal{L}$: the denoising loss we want to minimize
• $x$: the original, uncorrupted text sequence
• $\tilde{x}$: the corrupted input produced by applying the corruption function $c(\cdot)$ to $x$
• $\theta$: the model parameters (encoder and decoder weights)
• $P_\theta(x \mid \tilde{x})$: the probability the model assigns to the original text $x$ given the corrupted input $\tilde{x}$

The negative log maps the probability (a value between 0 and 1) into a loss: when the model assigns high probability to the correct reconstruction, the loss is low. When the model is uncertain or wrong, the loss is high. Minimizing this loss trains the model to reliably reconstruct clean text from corrupted inputs.

The choice of corruption function determines what the model must learn. Simple corruptions like random token replacement teach local dependencies. Complex corruptions like document rotation teach global structure. By combining multiple corruption types, we can train models that understand language at every level.

The encoder-decoder architecture fits naturally with denoising. The encoder processes the corrupted input bidirectionally, building rich representations. The decoder generates the original text autoregressively, learning to produce coherent output. This combination gives BART-style models the best of both worlds: bidirectional encoding for understanding and autoregressive decoding for generation. Notice that this architectural choice is not arbitrary: the bidirectional encoder is necessary because reconstructing missing information often requires looking ahead as well as back. The autoregressive decoder is necessary because generating coherent text requires conditioning each output token on all previously generated tokens.