Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Denoising Objectives › Document Rotation

Michael BrenndoerferUpdated July 14, 20254 min read

Language AI HandbookAll chapters

Part 22 of 67Pre-training Objectives

Chapter 161Denoising Objectives › Document Rotation

Reading section 6 of 15 · Document Rotation · 4 minSection 6 of 15 · 4 min

Section 6 of 15 · 4 min read

Denoising Objectives › Document RotationOne section · Sections in this chapter6 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Document rotation moves a random portion of the document from the beginning to the end, or vice versa. The text is "rotated" around a randomly chosen pivot point. This corruption breaks the document's beginning and end while preserving internal structure.

The motivation for document rotation is somewhat different from the other corruptions. Where token deletion and sentence permutation test reconstruction of missing or reordered content, document rotation specifically trains sensitivity to document boundaries. Many text genres have highly stereotyped openings and closings: news articles begin with a lede, academic papers begin with an abstract and introduction, fairy tales begin with "Once upon a time," and legal documents begin with formal declarations. By forcing the model to recognize when these expected openings are missing and to identify where the true beginning of a document lies, rotation trains awareness of genre conventions and global document structure.

In[18]:

Code

def apply_document_rotation(tokens):
    """
    Rotate the document by moving tokens from the beginning to the end.

    Args:
        tokens: List of tokens

    Returns:
        rotated: Tokens with rotation applied
        original: Original tokens for reconstruction target
        rotation_point: Where the rotation occurred
    """
    if len(tokens) < 2:
        return tokens, tokens, 0

    # Choose a random rotation point
    rotation_point = np.random.randint(1, len(tokens))

    # Rotate: move first part to the end
    rotated = tokens[rotation_point:] + tokens[:rotation_point]

    return rotated, tokens, rotation_point

In[19]:

Code

# Demonstrate document rotation
original_tokens = (
    "Once upon a time there was a brave knight who lived in a castle".split()
)
rotated, original, pivot = apply_document_rotation(original_tokens)

Out[20]:

Console

Original:
Once upon a time there was a brave knight who lived in a castle

Rotated at position 4 (29% into document):
there was a brave knight who lived in a castle Once upon a time

Moved to end: 'Once upon a time'

The rotated text starts mid-sentence with "was a brave knight..." rather than the natural opening "Once upon a time." The model must recognize this unnatural beginning and identify where the original document started. The classic fairy tale opening provides a strong signal about document structure, teaching the model to recognize common opening patterns like "Once upon a time," "In this paper," or "Chapter 1."

Start Token IdentificationLink Copied

BART uses a special approach for document rotation: it always starts the corrupted input at a sentinel token that marks where the original document began. This simplifies the task slightly but still requires the model to learn document structure.

Without the sentinel, the model must learn to identify document beginnings from content alone. Phrases like "In this paper," "Chapter 1," or narrative openings like "It was a dark and stormy night" signal document starts. The model develops sensitivity to these patterns through the rotation objective. In practice, the document rotation objective showed smaller benefits than text infilling and sentence permutation in the BART ablation studies. However, it is valuable for domains where document structure is highly stereotyped, such as scientific literature or news, and for tasks like document beginning detection and coherence scoring.

In[21]:

Code

def apply_document_rotation_with_sentinel(tokens, sentinel="[START]"):
    """
    Rotate document and mark original start position with sentinel.

    Args:
        tokens: List of tokens
        sentinel: Token to mark original document start

    Returns:
        rotated: Rotated tokens with sentinel marker
        original: Original tokens
    """
    if len(tokens) < 2:
        return tokens, tokens

    rotation_point = np.random.randint(1, len(tokens))
    rotated = tokens[rotation_point:] + [sentinel] + tokens[:rotation_point]

    return rotated, tokens

In[22]:

Code

rotated_with_sentinel, original = apply_document_rotation_with_sentinel(
    original_tokens
)

Out[23]:

Console

Original:
Once upon a time there was a brave knight who lived in a castle

Rotated with sentinel:
in a castle [START] Once upon a time there was a brave knight who lived

The sentinel token provides an explicit signal about where rotation occurred. The model's job becomes somewhat easier: generate the correct sequence given knowledge of where the original start was.