GPT-1 & GPT-2 (2018) › The Problem

By 2018, natural language processing research faced a fundamental challenge: the gap between task-specific model design and the desire for general-purpose language understanding. Most successful NLP systems required extensive task-specific engineering, with researchers designing specialized architectures and training procedures for each application. A model for sentiment analysis looked very different from a model for machine translation, which looked very different from a model for question answering. Each task demanded its own dataset, its own architectural choices, and its own training regimen.

This task-specific approach created several problems. First, it required large labeled datasets for each task, which were expensive and time-consuming to create. Training a sentiment analysis model required thousands of sentences annotated with sentiment labels. Building a question answering system required question-answer pairs carefully curated by human experts. Each new task demanded starting from scratch, collecting task-specific data, and designing task-specific architectures.

Second, the knowledge learned for one task did not transfer to other tasks. A model trained to classify text sentiment could not help with question answering. A model trained for named entity recognition could not assist with text summarization. This lack of transfer meant that each new application required investing in completely separate training processes, wasting the computational resources and knowledge gained from previous work.

Third, task-specific models often struggled with understanding language beyond their narrow training objectives. A sentiment classifier might accurately label movie reviews but fail to understand the sentiment implications in a news article or email. Models optimized for specific benchmarks often performed poorly when confronted with slightly different formulations of the same underlying task, or when applied in domains different from their training data.

The transformer architecture had shown promise, particularly in machine translation, but its application to other NLP tasks remained limited. Researchers were still designing task-specific transformer architectures, fine-tuning pretrained models on task-specific datasets, or using transformer components within larger task-specific systems. The idea that a single pretrained transformer could serve as a general-purpose foundation for multiple NLP tasks had not been convincingly demonstrated.

The field also lacked effective transfer learning approaches for NLP. While computer vision had successfully adopted transfer learning through ImageNet pretraining, where models pretrained on ImageNet could be fine-tuned for specific vision tasks, no comparable approach had emerged for language. Word embeddings like Word2Vec and GloVe had provided some transfer learning, but they were limited to word-level representations and didn't capture sentence-level or document-level understanding. The challenge was finding a pretraining objective and architecture that would learn general-purpose language representations useful across many downstream tasks.