Denoising Objectives › Sentence Permutation
Language AI HandbookAll chapters Denoising Objectives › Sentence PermutationOne section · Sections in this chapter5 / 15 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Sentence permutation shuffles the order of sentences within a document. While token shuffling breaks word-level order, sentence permutation breaks discourse-level structure. The model must learn how sentences relate to each other and what ordering makes a coherent document.
This corruption helps with tasks that require understanding document structure, such as summarization, document classification, and multi-document reasoning. A model that can reconstruct a permuted document must understand individual sentences and the logical and temporal relationships that bind them into a coherent whole. This is qualitatively different from token-level understanding because it requires reasoning at the paragraph level about causal links and how ideas contrast, elaborate, or form a narrative arc.
Think of sentence permutation as training the model to become a skilled editor who can look at a jumbled set of sentences and recognize which one should come first, which provides necessary background information, and which is a conclusion that only makes sense after earlier evidence is established. In practice, this skill transfers directly to tasks like multi-document summarization, where the model must understand which pieces of information are foundational and which are elaborations, even when those pieces arrive from different sources in no particular order.
def apply_sentence_permutation(text, sentence_delimiter="."):
"""
Randomly permute sentences within the document.
Args:
text: Input text string
sentence_delimiter: Character that separates sentences
Returns:
permuted: Text with sentences reordered
original: Original text for reconstruction target
"""
# Split into sentences (keeping delimiter)
sentences = [
s.strip() + sentence_delimiter
for s in text.split(sentence_delimiter)
if s.strip()
]
# Random permutation
permuted_order = np.random.permutation(len(sentences))
permuted_sentences = [sentences[i] for i in permuted_order]
return " ".join(permuted_sentences), text# Demonstrate sentence permutation
document = "The cat sat on the mat. It was a sunny day. The cat fell asleep. Later it woke up hungry."
permuted, original = apply_sentence_permutation(document)Original document: The cat sat on the mat. It was a sunny day. The cat fell asleep. Later it woke up hungry. Permuted document: Later it woke up hungry. It was a sunny day. The cat sat on the mat. The cat fell asleep. 4 sentences randomly reordered
The permuted document contains all the original information but in scrambled order. Notice how "Later it woke up hungry" appears without prior context about the cat falling asleep, making the narrative harder to follow. The model must learn to detect these coherence breaks and recognize that certain sentences require prior context to make sense.
Discourse Coherence
Sentence permutation forces the model to learn discourse markers and rhetorical relationships. Words like "however," "therefore," "first," and "finally" provide strong signals about sentence ordering. Pronoun resolution also provides clues: "it" in "It was a sunny day" must refer to something previously mentioned. When "It was a sunny day" appears as the first sentence of the permuted document, the model must recognize that this anaphoric reference is stranded without its antecedent. The temporal adverb "Later" in "Later it woke up hungry" signals that this sentence describes an event that follows some earlier event, so it cannot possibly be the opening sentence of the document.
Narrative text is harder because events unfold in a specific order. Scientific writing with clear logical progression is easier because explicit markers guide the ordering. Creative writing with complex temporal structure is hardest because the "correct" order may not be unique. In practice, sentence permutation pretraining is most beneficial for tasks involving multi-sentence reasoning, like abstractive summarization of long documents, dialogue generation, and narrative question answering. The BART authors found that sentence permutation, combined with text infilling, gave the best overall performance across multiple evaluation benchmarks, suggesting that learning both local and discourse-level reconstruction is complementary rather than redundant.
