Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

BART Pre-training › Sentence Permutation

Michael BrenndoerferOctober 19, 20253 min read

Language AI HandbookAll chapters

Part 25 of 67Encoder-Decoder Models

Chapter 181BART Pre-training › Sentence Permutation

Reading section 6 of 16 · Sentence Permutation · 3 minSection 6 of 16 · 3 min

Section 6 of 16 · 3 min read

BART Pre-training › Sentence PermutationOne section · Sections in this chapter6 / 16 ↓

Sections in this chapter16

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Moving from token-level to structural corruption, sentence permutation shuffles the order of sentences within a document. This tests the model's understanding of discourse coherence and document structure.aspects of language that token-level corruptions cannot address. Well-written documents have logical flow: ideas build on each other, references point backward to established concepts, and conclusions follow from premises. Sentence permutation disrupts this flow and asks the model to restore it.

How It WorksLink Copied

The document is first segmented into sentences (typically using punctuation heuristics or a sentence tokenizer, as covered in Part I). These sentences are then randomly permuted:

\tilde{x} = \operatorname{shuffle}([s_1, s_2, \ldots, s_k])

where:

$\tilde{x}$ : the corrupted document with reordered sentences
$[s_1, s_2, \ldots, s_k]$ : the original document as a sequence of $k$ sentences
$\text{shuffle}(\cdot)$ : a random permutation function that reorders the sentences

The shuffle operation creates a random permutation of the $k$ sentences, selecting uniformly from all $k!$ possible orderings. For a three-sentence document, there are $3! = 6$ possible orderings, and each has equal probability of being selected. For example, a three-sentence document might be reordered from $(s_1, s_2, s_3)$ to $(s_2, s_3, s_1)$ , placing what was the middle sentence first and the original first sentence last.

What the Model LearnsLink Copied

Sentence permutation teaches document-level coherence in several ways:

Discourse relations: How does one sentence logically follow another? Causal relationships ("Because X, therefore Y"), temporal sequences ("First X, then Y"), and elaboration patterns ("X. More specifically, Y") all provide cues for ordering. The model must recognize these implicit connections even when discourse markers are absent.
Referential coherence: Pronouns and definite references ("the man," "this idea") typically follow their antecedents. Permuted text disrupts these chains.a sentence might use "he" before the person has been introduced, or mention "this approach" before the approach has been described. The model learns to recognize and restore proper reference chains.
Topic flow: Well-organized text develops ideas progressively. Opening sentences introduce topics; middle sentences elaborate; closing sentences conclude or transition. The model learns to recognize these structural roles and restore this flow.

However, sentence permutation alone proved less effective than token-level corruption in BART's experiments. The learning signal is sparse (each training example provides only a few reordering decisions), and many permutations might be equally valid for loosely connected sentences. A document with three independent observations could be reasonably ordered in any sequence. This provides ambiguous signal to the model.

In[13]:

Code

import re


def sentence_permutation(
    text: str, sentence_end_pattern: str = r"(?<=[.!?])\s+"
) -> Tuple[str, str]:
    """Apply sentence permutation corruption.

    Returns:
        Tuple of (corrupted_text, original_text)
    """
    # Split into sentences
    sentences = re.split(sentence_end_pattern, text.strip())
    sentences = [s.strip() for s in sentences if s.strip()]

    # Shuffle sentences
    shuffled = sentences.copy()
    random.shuffle(shuffled)

    corrupted = " ".join(shuffled)
    return corrupted, text

In[14]:

Code

random.seed(42)

document = """Natural language processing has advanced rapidly. 
Transformers revolutionized the field in 2017. 
Today, large language models can perform diverse tasks."""

corrupted, original = sentence_permutation(document)

Out[15]:

Console

Original:
Natural language processing has advanced rapidly. 
Transformers revolutionized the field in 2017. 
Today, large language models can perform diverse tasks.

Corrupted (permuted):
Transformers revolutionized the field in 2017. Natural language processing has advanced rapidly. Today, large language models can perform diverse tasks.

The permuted version reads awkwardly.the chronological and logical flow is disrupted. The decoder must restore proper ordering based on discourse cues.