ALBERT › Sentence Order Prediction
Language AI HandbookAll chapters ALBERT › Sentence Order PredictionOne section · Sections in this chapter8 / 18 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
BERT's Next Sentence Prediction (NSP) task asked whether two sentences appeared consecutively in the original document. Critics argued this task was too easy: the model could often succeed by detecting topic consistency rather than understanding inter-sentence coherence.
ALBERT replaced NSP with Sentence Order Prediction (SOP), a harder task. Given two consecutive sentences, SOP asks whether they appear in the correct order or have been swapped.
A pretraining task where the model predicts whether two consecutive sentences appear in their original order or have been swapped. Unlike Next Sentence Prediction, SOP requires understanding discourse coherence since both sentences come from the same document and share the same topic.
def create_sop_examples(sentences, positive_ratio=0.5):
"""Create sentence order prediction training examples."""
examples = []
labels = []
for i in range(len(sentences) - 1):
sent_a = sentences[i]
sent_b = sentences[i + 1]
# Randomly decide: positive (correct order) or negative (swapped)
if np.random.random() < positive_ratio:
# Correct order
examples.append((sent_a, sent_b))
labels.append(1) # Is in correct order
else:
# Swapped order
examples.append((sent_b, sent_a))
labels.append(0) # Not in correct order
return examples, labels# Example sentences from a document
document_sentences = [
"The cat sat on the mat.",
"It was a sunny afternoon.",
"Birds chirped in the nearby tree.",
"The cat watched them lazily.",
"Eventually, it fell asleep.",
]
np.random.seed(42)
examples, labels = create_sop_examples(document_sentences, positive_ratio=0.5)Sentence Order Prediction Examples: ------------------------------------------------------------ Example 1 (Correct): Sentence A: The cat sat on the mat. Sentence B: It was a sunny afternoon. Example 2 (Swapped): Sentence A: Birds chirped in the nearby tree. Sentence B: It was a sunny afternoon. Example 3 (Swapped): Sentence A: The cat watched them lazily. Sentence B: Birds chirped in the nearby tree. Example 4 (Swapped): Sentence A: Eventually, it fell asleep. Sentence B: The cat watched them lazily.
SOP is harder than NSP because both sentences always come from the same document. The model cannot rely on topic mismatch to detect negatives. Instead, it must understand temporal and logical relationships between sentences. Does "It fell asleep" make sense before "It watched them lazily"? Answering that requires discourse understanding.
