BART Pre-training › Token Masking

Token masking is the most familiar corruption scheme, directly inherited from BERT's masked language modeling objective. Random tokens are replaced with a special [MASK] token, and the model must predict the original tokens. This approach has proven remarkably effective for learning contextual word representations, and BART includes it as one of its corruption options to benefit from these established strengths.

Given a sequence of tokens, we mask each token independently with probability $p$ (typically 15%, following BERT). The independence means that each token's fate is determined by its own random draw, without consideration of neighboring tokens:

$$\tilde{x}_i = \begin{cases} \text{[MASK]} & \text{with probability } p \\ x_i & \text{otherwise} \end{cases}$$

where:

• $\tilde{x}_i$: the $i$-th token in the corrupted sequence
• $x_i$: the $i$-th token in the original sequence
• $p$: the masking probability (typically 0.15)
• $\text{[MASK]}$: a special token showing a masked position

This piecewise definition captures a simple stochastic process: for each position $i$, flip a biased coin that lands on "mask" with probability $p$ and "keep" with probability $1-p$. The resulting corrupted sequence has the same length as the original, with [MASK] tokens scattered throughout. The specific positions vary each time we corrupt the same document. This provides varied training examples from identical source text.

For example, the sentence "The quick brown fox jumps over the lazy dog" might become "The [MASK] brown fox [MASK] over the lazy [MASK]". Notice that the structure remains intact. We can still see that something describes "brown," that an action connects "fox" to "over," and that some modifier precedes "dog." These structural clues are precisely what the model must learn to exploit.

Token masking teaches the model to use bidirectional context for word prediction. To predict "quick," the encoder must understand that something describing "fox" fits between "The" and "brown." The model learns that adjectives typically precede the nouns they modify, that certain adjectives co-occur (quick and brown both describe animals in motion), and that syntactic patterns constrain what words can appear where. This develops strong contextual word representations, the same capability that makes BERT effective for classification and extraction tasks where understanding word meaning in context matters most.

However, token masking has limitations that become apparent when we consider generation tasks. The corruption is always one-to-one: one mask corresponds to exactly one original token. This rigid correspondence means the model never learns to handle situations where the number of output tokens differs from the number of input tokens, an important skill for generation tasks. When summarizing, the model must compress many input tokens into fewer output tokens. When elaborating or explaining, it must expand. Token masking provides no training signal for these variable-length transformations.