GPT-3 › The Broader Context: Why 2020 Was the Right Moment

GPT-3 did not arrive in a vacuum. Its emergence was enabled by a convergence of factors that made 2020 the right time for this kind of model to exist and have the impact it did. Understanding this context helps explain why GPT-3 was so influential rather than just one more large language model in a long progression.

Available hardware had shifted decisively in favor of transformer training. NVIDIA's A100 GPU, released in 2020, offered roughly 20 petaFLOPS of bfloat16 performance per chip, dramatically faster than the V100s used for earlier large-scale training. Mixed-precision training techniques had matured, allowing models to train in 16-bit floating point with careful loss scaling while maintaining stability equivalent to 32-bit training. The combination meant that 175B parameter training was practically feasible for the first time, with OpenAI reportedly using a cluster of several thousand A100s for GPT-3's training run.

The software ecosystem had also matured. PyTorch and TensorFlow had both developed stable distributed training frameworks, and OpenAI had developed its own infrastructure for training models at this scale. Techniques like gradient checkpointing (trading compute for memory by not storing all activations during the forward pass) and pipeline parallelism (splitting the model across multiple machines along the depth dimension) made it possible to fit and train models that no single machine could hold in memory.

The dataset situation had evolved significantly from GPT-2's era. Common Crawl had been accumulating web content for years and provided a massive corpus of diverse text. The filtering techniques applied to Common Crawl, including deduplication, quality filtering using a trained classifier, and weighting by content quality metrics, represented a substantial advance over the simpler text collection methods of earlier models. The combination of raw scale and careful filtering produced training data that was both large enough and high-quality enough to train GPT-3 effectively.

Most importantly, the research community had developed the conceptual framework necessary to recognize and articulate what GPT-3's capabilities meant. The vocabulary of "in-context learning," "few-shot prompting," "emergent capabilities," and "scaling laws" did not exist before the GPT-3 paper. The paper itself introduced and defined these concepts. This provides a language for discussing phenomena that had been observed in less striking forms in earlier models but had not been systematically studied. The conceptual framing was as important as the empirical results in making GPT-3 so influential.