GPT-2 › GPT-2: Scaling Language Models for Zero-Shot Learning

In February 2019, OpenAI announced GPT-2 with an unusual strategy: they would not release the full model. The 1.5 billion parameter language model was, they claimed, too dangerous. It could generate text so convincingly human-like that they feared misuse, including fake news and spam as well as impersonation. The decision sparked fierce debate about responsible AI release, but it also pointed to a more basic shift. GPT-2 demonstrated that scale, applied to a simple autoregressive language modeling objective, could produce emergent capabilities that no one had explicitly trained for.

Think of GPT-2 as a machine that reads 40 gigabytes of the internet, then attempts to write more of it. It learns not by being told what tasks exist or how to solve them, but by absorbing the patterns of human communication across millions of web pages. When a person posts a question on the internet, someone answers it. When an article headline appears, a body of text follows. When someone writes a passage in French, more French text comes after. GPT-2 learns all of these patterns simultaneously, and the astonishing result is that it can reproduce any of them on demand, even without being explicitly told which pattern is needed.

This chapter examines GPT-2's contributions to language modeling. We'll explore how OpenAI scaled up from GPT-1's 117 million parameters across four model sizes, understand the architectural refinements that enabled stable training at scale, analyze the WebText dataset that taught the model from high-quality internet text, and investigate the zero-shot learning phenomenon that made GPT-2 far more than just a text generator.

The key insight that runs through every section of this chapter is that GPT-2 did not change the basic objective from GPT-1. It still trained by predicting the next token. What changed was everything else: how much data, how large the model, and how carefully the training was engineered to remain stable at scale. These changes in quantity produced changes in quality that surprised even the researchers who built the system.

By the end of this chapter, you'll understand why GPT-2 marked the transition from fine-tuned specialists to general-purpose language models. You'll see the precise architectural choices that made large-scale stable training possible, appreciate the subtle genius of training from naturally occurring task demonstrations embedded in web text, and have a working implementation of the GPT-2 architecture in PyTorch.