GPT-1 & GPT-2 (2018) › 2018: GPT-1 & GPT-2

OpenAI's introduction of GPT-1 in 2018 and GPT-2 in 2019 changed natural language processing. This showed that large-scale autoregressive language models pretrained on vast text corpora could achieve remarkable performance across diverse NLP tasks without task-specific architectures. The GPT models, which stood for "Generative Pre-trained Transformer," marked a departure from the task-specific fine-tuning approaches that had dominated NLP research. Instead, they showed that a single generative language model could be pretrained on unlabeled text data and then adapted to various downstream tasks through transfer learning, fundamentally changing how researchers approached NLP problems.

By 2018, the transformer architecture had proven its effectiveness in machine translation, but most NLP applications still required task-specific architectures and training procedures. Researchers would train separate models for tasks like question answering, sentiment analysis, and text classification, each requiring labeled datasets and careful architectural choices. The idea that a single pretrained model could serve as a foundation for multiple tasks had been explored in computer vision with ImageNet pretraining, but had not yet been demonstrated effectively for language understanding.

GPT-1 emerged from OpenAI's recognition that the transformer architecture's ability to process variable-length sequences and capture long-range dependencies made it an ideal foundation for building general-purpose language representations. The model was pretrained on a large corpus of unlabeled text using a simple objective: predict the next word given all previous words. This autoregressive pretraining task, while seemingly simple, forced the model to develop internal representations of linguistic patterns and contextual meaning that proved useful for downstream applications.

The breakthrough came from demonstrating a new paradigm for NLP: pretraining followed by task-specific fine-tuning. GPT-1 showed that a model pretrained on a diverse corpus could be fine-tuned on specific tasks with minimal task-specific modifications, achieving strong performance with far less labeled data than would be required to train task-specific models from scratch. This transfer learning approach would become the standard methodology for NLP research and development.

GPT-2, released less than a year later, took this paradigm to unprecedented scale and demonstrated surprising emergent capabilities. With 1.5 billion parameters trained on an even larger and more diverse text corpus, GPT-2 showed that simply scaling up autoregressive language models could produce systems with impressive zero-shot capabilities. The model could perform reading comprehension and question answering, along with translation and summarization, without any task-specific training, simply by conditioning on task descriptions or examples provided in the input text. This demonstration of emergent capabilities from scale would foreshadow the remarkable developments in large language models that would follow in subsequent years.