BART Pre-training › Document Rotation

Document rotation is the most aggressive structural corruption in BART's repertoire. A random token is selected as the new starting point, and the document is rotated so that it begins with that token. Unlike sentence permutation, which at least preserves sentence boundaries, document rotation can split sentences arbitrarily, placing sentence fragments at the beginning or end.

Given a document with tokens $[x_1, x_2, \ldots, x_n]$, we uniformly sample a position $k \in \{1, \ldots, n\}$ and rotate:

$$\tilde{x} = [x_k, x_{k+1}, \ldots, x_n, x_1, x_2, \ldots, x_{k-1}]$$

where:

• $\tilde{x}$: the rotated (corrupted) document
• $x_1, x_2, \ldots, x_n$: the original document with $n$ tokens
• $k$: the randomly chosen rotation point (sampled uniformly from 1 to $n$)
• $[x_k, \ldots, x_n]$: tokens from position $k$ to the end become the new beginning
• $[x_1, \ldots, x_{k-1}]$: tokens before position $k$ are moved to the end

Think of the document as arranged in a circle. We pick a random position and "cut" the circle there, then lay it out linearly starting from that cut point. All tokens remain in their relative order, but the absolute positions shift. The original beginning might now appear in the middle or at the end.

For example, "The quick brown fox jumps" with $k=3$ becomes "brown fox jumps The quick." The fragment "brown fox jumps" appears first, followed by "The quick".a clear violation of sentence structure that signals something is wrong.

Document rotation teaches the model to identify document boundaries, specifically where a coherent document begins. This seems like a narrow skill, but it has practical uses:

• Document structure recognition: Real documents have clear beginnings (titles, topic sentences) and endings (conclusions, sign-offs). The model learns to recognize these structural markers (capitalized first words, sentence-initial positions, opening phrases like "In this paper" or "The purpose of") that distinguish true document starts from arbitrary mid-document positions.

• Global coherence: Unlike sentence permutation, which preserves some local structure, rotation can place mid-sentence fragments at document boundaries. The model must recognize incomplete thoughts, such as a sentence missing its subject or a phrase without context, as evidence of artificial boundaries.

In practice, document rotation was the least effective corruption scheme when used alone. The task is difficult (many rotation points produce plausible-looking starts, especially in long documents where sentence-initial positions are common), and the learning signal is limited to identifying a single boundary per document. The model receives one bit of useful information per training example: where the true start is.