RoBERTa › More Data, Longer Training

BERT was trained on BookCorpus (800M words) and English Wikipedia (2,500M words), totaling about 16GB of uncompressed text. RoBERTa expanded this substantially by adding three more datasets:

• CC-News: 76GB of news articles crawled from Common Crawl
• OpenWebText: 38GB of web content from Reddit-linked URLs
• Stories: 31GB of story-like content from Common Crawl

The combined dataset is roughly 160GB, ten times larger than BERT's original training data. But data alone isn't enough. RoBERTa also trained for significantly more steps.

The choice of datasets is deliberate. BERT's original training corpus was relatively narrow: books and Wikipedia represent careful, edited prose. They are high quality but also stylistically homogeneous. Real-world text varies widely in form. News articles use different syntactic patterns than Wikipedia. Online forum posts use different vocabulary than books. Stories use different narrative structures than all of the above. By adding CC-News alongside OpenWebText and Stories, RoBERTa's training corpus covers a much broader range of linguistic variation.

This breadth matters for downstream performance. When you fine-tune a pretrained model on a task like sentiment analysis or question answering, your task data may come from a domain that looks more like news or web text than like Wikipedia. A model pretrained exclusively on edited prose will have representations that are sharper for that style but less reliable across registers. The broader pretraining corpus in RoBERTa makes it more domain-agnostic and therefore more useful across a wider range of applications.

The OpenWebText dataset deserves special mention because of its curation methodology. Rather than crawling web pages at random, the creators filtered Common Crawl pages by whether they were linked from Reddit posts that received at least three upvotes. The logic is that Reddit's voting mechanism is a crude quality filter: links that attracted engagement tend to point to content that humans found interesting and readable. This approach yields a corpus that is web-scale in size but somewhat higher in quality than random web scrapes, avoiding spam and boilerplate along with the garbled text that plagues unfiltered web corpora.

The paper systematically varied training duration to understand its impact. They found that longer training consistently improved performance, even past the point where loss on the training data plateaued. This suggests that the model continues to learn useful representations even when the pretraining objective stops improving.

This finding is counterintuitive if you think about training purely in terms of loss minimization. The naive picture would say: once the loss stops decreasing, you've extracted all the learning signal available, and further training is wasteful. But downstream task performance tells a different story. The model keeps improving on MNLI, SST-2, and other tasks even after the pretraining loss has flattened.

The explanation lies in what the pretraining loss measures versus what fine-tuning tasks require. The MLM loss on a large diverse corpus is a noisy proxy for the quality of internal representations. Two models with nearly identical MLM perplexity can have quite different representations for the same sentences, depending on which aspects of language structure they have internalized. Fine-tuning tasks expose these representational differences. The model that has seen more data and trained longer tends to have more consistent, transferable representations even when the difference in raw MLM loss is small.

Think of it as the difference between a student who has studied just enough to pass an exam and one who has studied until the subject feels natural. Both might score similarly on a closed-book exam on the studied material, but the second student will do much better when asked to apply the knowledge in a novel context. Longer pretraining builds the second kind of understanding.