GPT-1 & GPT-2 (2018) › The Solution

GPT-1 introduced a simple but powerful approach: pretrain a transformer-based language model on a large unlabeled text corpus using an autoregressive objective, then fine-tune the same model on downstream tasks with minimal task-specific modifications. This approach showed that the knowledge gained during pretraining could be effectively transferred to various NLP applications, dramatically reducing the need for task-specific data and architectures.

GPT-1 used the transformer decoder architecture, which had proven effective for sequence generation tasks. Unlike BERT, which used the transformer encoder with bidirectional attention, GPT-1 used only the decoder stack with unidirectional (left-to-right) attention. This unidirectional attention meant that each position could attend to previous positions but not future positions, making the architecture autoregressive and suitable for text generation.

The transformer decoder stack consisted of multiple layers, each containing self-attention mechanisms and feedforward networks. Self-attention allowed each position to attend to all previous positions, enabling the model to capture long-range dependencies while maintaining the autoregressive property needed for generation. The multi-head attention mechanism used in GPT-1 allowed the model to attend to different types of relationships simultaneously, while layer normalization and residual connections helped with training stability and gradient flow.

The pretraining objective was elegantly simple: given a sequence of tokens, predict the next token. More formally, given a sequence of tokens $x_1, x_2, \ldots, x_n$, the model learned to maximize the likelihood $P(x_i | x_1, \ldots, x_{i-1})$ for each position $i$. This autoregressive objective forced the model to learn rich representations of language structure, because accurate next-word prediction required syntax and contextual meaning together with world knowledge implicit in the preceding text.

The pretraining corpus for GPT-1 consisted of a large collection of text from books, which provided diverse and relatively high-quality language data. The model learned from millions of sentences across many topics and styles, developing internal representations that captured linguistic patterns, semantic relationships, and factual knowledge. The diversity of the training data ensured that the learned representations would be general rather than task-specific.

The key innovation was in the fine-tuning procedure. Rather than starting from random weights for each downstream task, GPT-1 took the pretrained model and fine-tuned it on task-specific labeled data. For classification tasks, the model would add a linear classification layer on top of the final transformer layer. For sequence-to-sequence tasks, the model could be adapted with additional layers or modified input formats. The fine-tuning process updated all model parameters, allowing the pretrained knowledge to adapt to the specific requirements of each task while preserving the general language understanding learned during pretraining.

The transfer learning effectiveness came from the pretrained model having already learned useful representations of language. The lower layers of the transformer captured low-level features like word order and syntax, while higher layers captured more abstract semantic relationships. When fine-tuning on a downstream task, these pretrained representations provided a strong foundation, allowing the model to achieve good performance with far less task-specific data than would be needed to train from scratch.

GPT-2 scaled up this approach dramatically. Released in 2019, GPT-2 increased the model size from GPT-1's 117 million parameters to 1.5 billion parameters across multiple variants. The training corpus was also expanded, incorporating a much larger and more diverse collection of text scraped from web pages. This web scrape, known as WebText, contained billions of words across diverse domains and writing styles. It therefore represented a broader sample of human language use.

The architecture of GPT-2 was largely similar to GPT-1, but with important refinements. The model used layer normalization placed at the beginning of each sub-layer rather than at the end, improving training stability. GPT-2 also used a modified initialization scheme that scaled residual connections by a factor of $1/\sqrt{n}$ where $n$ is the number of layers, helping with training deeper networks. The vocabulary size was expanded to accommodate the more diverse training corpus.

Perhaps the most significant departure in GPT-2 was the emphasis on zero-shot learning. While GPT-1 focused on demonstrating transfer learning through fine-tuning, GPT-2 showed that sufficiently large autoregressive language models could perform tasks without any task-specific training at all. By conditioning on natural language task descriptions or examples, GPT-2 could perform translation and summarization as well as question answering without fine-tuning.

The zero-shot capability emerged from the model's training objective. During pretraining, the model learned to continue text sequences, which meant it learned patterns of how language describes tasks and how examples are formatted. When given a prompt like "Translate English to French: sea otter => loutre de mer", the model could recognize the task description and example format, then generate appropriate translations for subsequent examples. WebText likely contained many examples of task instructions and structured text, including translations, which helped the model learn these patterns implicitly.