BART Pre-training › Text Infilling

Text infilling is BART's most important corruption scheme and the one that proved most effective in experiments. Instead of corrupting individual tokens, we corrupt contiguous spans of varying lengths. Critically, each span is replaced with a single mask token regardless of how many tokens it originally contained. This creates a challenging reconstruction task that teaches the model key skills for generation: predicting what content is missing and how much content is missing.

The infilling process has two steps that together create spans of varying lengths positioned throughout the document:

1. Sample span lengths: Draw span lengths from a Poisson distribution with parameter $\lambda$ (typically $\lambda = 3$). This creates spans of varying sizes: many single-token spans, some two-token spans, fewer three-token spans, and so on. The Poisson distribution provides a natural way to generate this variety with a single parameter controlling the expected length.

2. Sample span positions: Randomly select starting positions for spans until approximately $p$ of the total tokens are covered by spans. The sampling continues until the target corruption level is reached. This keeps consistent corruption intensity across documents of different lengths.

Each span, regardless of length, is replaced with exactly one [MASK] token:

$$\text{span}(x_i, x_{i+1}, \ldots, x_{i+k}) \to \text{[MASK]}$$

where:

• $x_i, x_{i+1}, \ldots, x_{i+k}$: a contiguous span of $k+1$ tokens starting at position $i$
• $\text{[MASK]}$: a single mask token that replaces the entire span
• $\to$: indicates the replacement operation

This many-to-one mapping is the important innovation. A single [MASK] in the input might correspond to zero tokens, one token, three tokens, or even more in the output. The model cannot simply count masks and produce the same number of tokens. It must learn to judge, from context alone, how extensive each gap is.

For example, "The quick brown fox jumps over the lazy dog" might become "The [MASK] fox [MASK] the [MASK] dog" where:

• "[MASK]₁" replaces "quick brown" (2 tokens)
• "[MASK]₂" replaces "jumps over" (2 tokens)
• "[MASK]₃" replaces "lazy" (1 token)

The decoder must figure out that the first mask needs two tokens (an adjective pair), the second needs two (a verb phrase), and the third needs one (a single adjective), all from the contextual clues in "The ... fox ... the ... dog."

Why use Poisson? We want span lengths that vary naturally, mostly short spans with occasional longer ones, without needing to set a maximum length. The Poisson distribution is ideal because it models counts of independent events and has a single parameter $\lambda$ controlling the average. This is convenient: rather than specifying an entire distribution over possible lengths, we tune one number and get a reasonable distribution automatically.

The Poisson distribution gives the probability of observing exactly $k$ events:

$$P(k; \lambda) = \frac{\lambda^k e^{-\lambda}}{k!}$$

where:

• $P(k; \lambda)$: the probability of observing exactly $k$ events (here, a span of $k$ tokens)
• $k$: the span length (number of tokens, can be 0, 1, 2, ...)
• $\lambda$: the expected (average) span length, set to 3 in BART
• $e^{-\lambda}$: the exponential decay factor so probabilities sum to 1
• $k!$: factorial of $k$, which appears because ordering doesn't matter within the span count

The formula works as follows: $\lambda^k$ makes longer spans increasingly likely as $\lambda$ grows.if the average is higher, we should see more long spans. Meanwhile, $k!$ in the denominator penalizes very long spans (since factorial grows faster than exponential), preventing the distribution from placing too much mass on extremely long spans that would destroy too much context. The $e^{-\lambda}$ term normalizes everything to sum to 1. This keeps we have a valid probability distribution. Together, these three components create a distribution peaked near $\lambda$ with a long tail toward higher values.

With $\lambda = 3$:

• $P(0) \approx 0.05$: 5% of spans have length 0 (insertions!)
• $P(1) \approx 0.15$: 15% single-token spans
• $P(2) \approx 0.22$: 22% two-token spans
• $P(3) \approx 0.22$: 22% three-token spans
• $P(\geq 4) \approx 0.36$: 36% longer spans

The inclusion of zero-length spans is particularly clever and deserves special attention. A zero-length span means inserting a [MASK] token between two existing tokens without removing anything. The decoder must learn to generate empty content for this mask.to recognize that sometimes a mask is a false alarm requiring no output. This teaches the model that not every mask needs to expand into content, a nuance absent from standard masking approaches where every mask always corresponds to exactly one token.

Text infilling develops generation capabilities that transfer directly to downstream tasks:

1. Length prediction: The model must infer how many tokens each mask represents. A single mask might expand to zero, one, five, or more tokens. This requires understanding semantic and syntactic constraints.

2. Coherent multi-token generation: Generating "quick brown" as a unit requires understanding that these words form a coherent phrase modifying "fox." The model cannot generate each token independently.

3. Span boundary detection: The model must determine where masked content ends and original content resumes. When generating tokens for a mask, the model must decide when to stop generating.

This directly addresses a limitation of T5's span corruption. T5 uses sentinel tokens that indicate span boundaries and generates only the corrupted spans with their sentinels. BART must reconstruct the full original text, interleaving generated and copied content smoothly. This trains the model for generation tasks where the output blends novel phrases with material from the input. This is exactly what happens in summarization and paraphrasing, among other applications.