GPT-1 › Summary

GPT-1 introduced a powerful recipe for language understanding: pre-train a transformer decoder on large-scale text using next-token prediction, then fine-tune on downstream tasks with minimal architectural changes. The key contributions and takeaways include:

• Unified architecture: A single 12-layer transformer decoder handles both pre-training and diverse downstream tasks. The same model structure enables text generation and classification.

• Generative pre-training: The simple objective of predicting the next token, applied at scale to BooksCorpus, produces representations rich enough for transfer learning across classification and entailment, along with similarity and question answering.

• Input transformation: Different tasks are reformulated as sequences with special delimiters ([Start], [Delim], [Extract]), allowing the same model to process various input formats.

• Auxiliary objectives: Including language modeling loss during fine-tuning ($\lambda = 0.5$) improves transfer by regularizing against forgetting.

• Transfer across tasks: Pre-training on narrative text transferred to formal reasoning tasks (RACE, COPA). This shows that language modeling captures general linguistic competence.

GPT-1 set the stage for the scaling revolution that followed. GPT-2 scaled the approach 10x, GPT-3 scaled it 1000x, and subsequent models have pushed further still. But the core insights, decoder-only architecture, pre-training on raw text, fine-tuning for tasks, remain foundational to how we build language AI today.