RoBERTa › Removing Next Sentence Prediction

BERT was trained with two objectives: Masked Language Modeling (MLM) and Next Sentence Prediction (NSP). The NSP task classifies whether two sentences appear consecutively in the original text or are randomly paired. The intuition was that NSP would help the model understand document-level coherence. Tasks like question answering and natural language inference require understanding how sentences relate to one another, and the hope was that training the model to explicitly predict sentence adjacency would instill this capability.

RoBERTa's first major finding was that NSP hurts more than it helps. When the researchers trained BERT with MLM only, removing NSP entirely, performance improved on most downstream tasks.

Why would a seemingly useful objective hurt performance? Several factors contribute:

The NSP task is too easy. Randomly sampled sentences often come from completely different documents and topics. The model can solve NSP by detecting topic shifts rather than learning discourse coherence. A sentence about cooking paired with one about astrophysics is trivially distinguishable without understanding narrative flow.

NSP constrains the input format. BERT's NSP training requires inputs to be sentence pairs, typically short. This prevents the model from seeing longer contiguous passages that might teach it more about extended context and document structure.

NSP dilutes the learning signal. Half the model's training updates come from a task that may not transfer well to downstream applications. Those gradients could instead strengthen the MLM objective.

The deeper problem with NSP is that it was designed to solve a real problem but implemented in a way that created a shortcut. When you construct negative examples for a binary classification task by randomly sampling from the entire dataset, you inadvertently make the negative examples very easy to identify, because random sentences drawn from a large corpus tend to differ in topical content and use different vocabulary or styles from the preceding sentence. A model that simply memorizes the surface statistics of topical continuity can solve NSP without learning anything about discourse coherence. This is a common failure mode in auxiliary task design: the training signal teaches the model to solve a proxy task through a path that doesn't generalize.

In practice, the removal of NSP also simplifies the training pipeline considerably. BERT's NSP format requires storing sentence boundaries, constructing 50/50 positive-negative pair samples, and tracking two separate losses during training. Dropping NSP means the model processes plain tokenized text with no additional bookkeeping. The training code becomes simpler and less prone to data preparation bugs.

Subsequent work has explored whether some form of sentence-level objective is still beneficial. ALBERT, which we'll examine in a later chapter, replaced NSP with a harder Sentence Order Prediction task that forces the model to distinguish the correct ordering of two sentences from the reversed ordering. Because both sentences in the ALBERT task come from the same document, topic shortcuts don't work, and the model must learn something about discourse flow. The ALBERT experiments suggest that NSP's failure was not about the idea of sentence-level objectives, but about the poor quality of the negative examples BERT used.

The RoBERTa paper tested several input formats:

The DOC-SENTENCES format performed best. Models trained without NSP and on longer contiguous passages learn better representations. The takeaway: NSP was a red herring. The simple MLM objective on longer contexts works better.

The FULL-SENTENCES and DOC-SENTENCES formats differ in one subtle but important way: DOC-SENTENCES never crosses document boundaries within a single input sequence. If you're near the end of a document, you stop there and pad the sequence rather than beginning a new document mid-sequence. The rationale is that different documents may have stylistic inconsistencies, genre differences, or topical discontinuities that could confuse the model. By keeping sequences document-local, you ensure that all the context the model sees for a given token comes from the same coherent source. The empirical evidence from the RoBERTa ablations supports this: DOC-SENTENCES edges out FULL-SENTENCES by a small but consistent margin.

In practice, the difference between FULL-SENTENCES and DOC-SENTENCES is small enough that many downstream implementations use FULL-SENTENCES for simplicity. When your documents are long and uniform, the formats are nearly equivalent. When your corpus contains very short documents, DOC-SENTENCES may result in more padding and less efficient training, at which point the theoretical advantage of avoiding document boundary crossings becomes a practical cost.