GPT-2 › Summary

GPT-2 demonstrated that scaling autoregressive language models yields emergent capabilities beyond next-token prediction. The key contributions include:

• Model family: Four sizes from 117M to 1.5B parameters, letting systematic study of scaling behavior while keeping architecture constant. This design pattern became the template for virtually all subsequent large language model releases.

• Architectural refinements: Pre-normalization for stable training at depth, GELU activation for smoother gradients and better gradient flow through slightly negative activations, and modified initialization to control variance growth across deep residual networks.

• WebText training: 40 GB of text from Reddit-filtered web pages. This provides diverse, naturally-occurring task demonstrations without explicit supervision. The internet's natural embedding of question-answering and summarization, along with translation patterns, was the implicit curriculum that enabled zero-shot transfer.

• Zero-shot capabilities: Reading comprehension and summarization, along with translation, emerged from pure language modeling, achieved through prompting that aligned test-time queries with training-time text patterns. The model never learned these tasks explicitly; it absorbed them through statistical regularity in the training data.

• Generation quality: Coherent multi-paragraph text generation that, with appropriate sampling strategies (nucleus sampling, temperature tuning), could pass for human-written. The 1,024-token context window and transformer's direct attention mechanism were the technical enablers of this coherence.

• Scaling laws foreshadowed: The consistent power-law relationship between model size and perplexity across multiple benchmarks presaged the formal scaling laws that Kaplan et al. would publish in 2020, establishing that capability improvements are predictable functions of compute and data.

GPT-2 marked the transition from fine-tuned specialists to general-purpose language models. Its limitations around factual reliability and bias, along with weak long-range coherence, remain active research areas. But its core insight, that scale enables emergence, fundamentally reshaped expectations for what language models could become. In the next chapter, we'll examine GPT-3, which took this insight to its logical conclusion by scaling to 175 billion parameters and finding that few-shot learning, when the model is large enough, can match or exceed fine-tuned performance on many benchmarks.