GPT-2 › WebText: Learning from the Internet

GPT-2's training data represented a significant departure from previous approaches. Rather than using carefully curated datasets like BooksCorpus (used for GPT-1) or Wikipedia, OpenAI created WebText by scraping outbound links from Reddit posts with at least 3 upvotes. This might sound like a small methodological detail, but it shaped what the model learned.

The motivation behind this shift was scale and diversity. Books and Wikipedia are high-quality but limited. Books tend toward certain narrative styles and genres. Wikipedia is encyclopedic and factual but covers only a fraction of human knowledge and communication. The internet, for all its chaos, contains text in every genre and style, across registers and domains. By training on web text filtered for some quality signal, the researchers could access training data orders of magnitude larger than curated datasets while still avoiding the worst content.

The WebText construction process involved several key decisions, each which reflects a deliberate choice about the tradeoff between data quantity and quality:

• Source selection: Links from Reddit with $\geq 3$ karma (upvotes minus downvotes) were collected. This crowdsourced filtering biased toward content that humans found interesting, informative, or entertaining. A Reddit post linking to random spam is unlikely to accumulate three upvotes; a link to an interesting news article, a thoughtful opinion piece, or a useful tutorial is much more likely to be shared. Reddit thus served as a massive, distributed quality-filtering system for the entire web.

• Deduplication: Near-duplicate documents were removed to prevent the model from memorizing repeated content. This is important for model quality: a model trained on millions of copies of the same article will reproduce that article well but generalize poorly.

• Minimal preprocessing: Unlike many previous datasets, WebText preserved formatting and punctuation, along with case. The model learned from text as it appears naturally on the web, including code and tables as well as quoted material. This choice meant the model saw a wider variety of text styles, which likely contributed to its versatility.

• Exclusion of Wikipedia: Wikipedia was deliberately excluded to enable fair evaluation on Wikipedia-based benchmarks. Including it would have made it impossible to evaluate the model's ability to generalize on any test derived from Wikipedia content.

This data collection strategy reflected a key insight: the internet contains a vast range of text. By learning from this diversity, GPT-2 could potentially generalize to many tasks without task-specific training. The internet also contains implicit task demonstrations: question-and-answer threads, translation examples, text followed by summaries, arguments followed by counterarguments. The model learned these patterns not by being taught them but by absorbing millions of naturally occurring examples.

The WebText approach changed what followed for what the model could do. First, it scaled easily since the internet provides virtually unlimited text. Second, the diversity exposed the model to many natural "tasks" embedded in web text: answering questions, summarizing articles, translating between languages, and explaining concepts. Third, it introduced biases present in Reddit's user base and the content they share. Reddit's demographic skews toward English-speaking, technically educated, younger users, and this bias became part of GPT-2's worldview.

A subtle but important point is that WebText does not just provide language patterns. It provides implicit task demonstrations. Consider what kinds of text appear naturally on the internet: an article followed by a comments section where users summarize or critique it teaches summarization. A Stack Overflow question followed by a correct answer teaches question answering. A news article in English followed by its translation teaches translation. A recipe followed by user reviews teaches evaluation. The model absorbs all of these as instances of next-token prediction, but in doing so it learns the structure of each task type. This is why the zero-shot capabilities described in the next section are not magic: they are the expected consequence of training on text that contains millions of implicit task demonstrations.