GPT-1 › GPT-1

In June 2018, OpenAI released a paper titled "Improving Language Understanding by Generative Pre-Training." The model it introduced, later known as GPT-1, wasn't the largest language model of its time, nor did it immediately dominate benchmarks. But it showed that a simple recipe of unsupervised pre-training on raw text followed by supervised fine-tuning could achieve state-of-the-art results across diverse NLP tasks. This "pre-train then fine-tune" approach would become the dominant paradigm for natural language processing, reshaping how the entire field thought about model development.

GPT-1 arrived as the field was changing. Researchers knew that language models trained on large corpora learn useful representations, but the common view held that task-specific architectures were necessary for each downstream application. If you wanted to classify sentiment, you built a sentiment classifier. If you wanted to detect entailment, you designed an entailment model. Every new task required designing a new system from scratch, curating labeled data, and training it end-to-end. GPT-1 challenged this assumption. By using a single, unified transformer decoder architecture for both pre-training and fine-tuning, it showed that generative pre-training creates representations detailed enough to transfer to classification and similarity tasks, as well as entailment and question answering, with minimal architectural changes.

Think of GPT-1's pre-training as building a powerful general-purpose language engine. Just as a car engine can power vehicles of many different designs, the pre-trained transformer provides a shared foundation that can be adapted to many different downstream tasks. You don't build a new engine for each car model: you refine an existing one. GPT-1 extended this logic to language: don't build a new model for each NLP task, pre-train a shared representation and adapt it through fine-tuning.

The model was trained on the BooksCorpus dataset, roughly 800 million words spanning thousands of unpublished novels across genres. This raw, unlabeled text provided a rich training signal without any human annotation. The key insight behind the approach was that predicting the next word in a novel requires deep language understanding: you need to track characters across chapters, understand discourse coherence, resolve ambiguous pronoun references, and reason about causal and temporal relations. A model forced to predict narrative text reliably will, in the process, develop broadly useful linguistic representations.

What made GPT-1 particularly significant was its architectural simplicity. The model used a decoder-only transformer, a straightforward stack of self-attention layers with causal masking, applied uniformly at both pre-training and fine-tuning time. There were no task-specific components, no complex multi-task training objectives, and no elaborate engineering. The simple recipe was easy to reproduce and extend to larger models. This last property would prove decisive: GPT-2, GPT-3, and their successors all follow the same fundamental design, simply larger.

This chapter explores the GPT-1 architecture in detail. We'll examine its decoder-only design, understand the pre-training objective and data, walk through the fine-tuning approach that enabled transfer learning, and assess the model's impact on the trajectory of NLP research. Along the way, we'll implement the core components in PyTorch to build concrete intuition for how the system works from the ground up.