GPT-1 › Transfer Learning Results

GPT-1 demonstrated strong transfer learning across 12 diverse NLP tasks. The pre-training on BooksCorpus, despite never seeing task-specific supervision, produced representations that transferred effectively to classification and similarity tasks, as well as question answering. This result validated the central hypothesis of the paper: that a model trained purely to predict text can develop general-purpose language representations without any explicit semantic annotation.

The breadth of the transfer was striking. The same pre-trained model, fine-tuned separately, improved over the state of the art on sentiment classification (a relatively straightforward task), natural language inference (which requires logical reasoning), semantic textual similarity (which requires understanding paraphrase and meaning equivalence), commonsense reasoning (COPA), and multi-paragraph reading comprehension (RACE). These tasks were sufficiently different that it would not have been obvious, prior to GPT-1, that a single set of pre-trained representations could serve all of them well. The diversity of the gains was arguably more significant than the magnitude, because it suggested that the pre-trained representations captured something fundamental about language rather than patterns specific to any one task type.

The pattern of improvements also revealed something about what pre-training provides. The largest gains appeared on tasks with limited training data (RACE with 87,866 examples, COPA with 400 training examples) and on tasks requiring reasoning over longer contexts (reading comprehension). Tasks where large training sets already existed showed smaller improvements, consistent with the view that pre-training is most valuable when labeled data is scarce. When you have millions of labeled examples, supervised learning alone can learn the relevant patterns. When you have only hundreds or thousands, the pre-trained representations provide a strong initialization that compensates for data scarcity.

The following table summarizes GPT-1's performance compared to previous state-of-the-art models at the time of publication (June 2018):

The improvements were particularly dramatic on tasks with limited training data. RACE, a reading comprehension dataset, saw a 14.9 point improvement. CoLA, a grammatical acceptability task, improved by 10.4 points. These gains suggest that pre-training captures linguistic knowledge that is difficult to learn from small supervised datasets alone.

Let's examine how the pre-trained model transfers knowledge. The key question is: what does the model learn during pre-training that helps with downstream tasks? The answer is not a single thing but a hierarchy of linguistic capabilities that emerge at different layers of the network. Research on probing classifiers, which train simple linear models on the representations from each layer to predict linguistic properties, has revealed a consistent pattern: lower layers capture local syntactic features like part-of-speech tags and constituent boundaries, while higher layers capture more abstract semantic and discourse properties. This hierarchical organization parallels what has been observed in convolutional neural networks for images, where earlier layers detect edges and textures while later layers detect object parts and semantic categories.

The GPT-1 paper's own ablation studies examined what happens when you fine-tune only the top $k$ layers and keep the bottom $12 - k$ layers frozen. Performance improves monotonically with $k$: using all 12 layers beats using 11, which beats using 10, and so on. This means every layer contributes useful pre-trained knowledge to downstream performance, and there is no clear cutoff where the lower layers stop contributing. The implication is that the full depth of the pre-trained representation is relevant for transfer; top-level semantic features alone are insufficient.

The diminishing-returns pattern in the ablation also makes intuitive sense. The first few layers of pre-training produce enormous gains because they capture basic syntactic and semantic regularities that are useful for almost any NLP task. The later layers contribute more specialized discourse and reasoning capabilities that are valuable but less universally necessary. This pattern suggests that if compute is limited, the most cost-effective strategy is to invest in pre-training depth up to a point, then accept diminishing returns on additional layers.

The figure shows simulated data based on patterns from the GPT-1 paper's ablation studies. Key observations:

• All layers contribute: Each additional pre-trained layer improves performance
• Diminishing returns: The marginal benefit decreases as more layers are added
• Task variation: Some tasks (like SST-2) benefit more from deeper features than others

The GPT-1 paper included several ablation studies that revealed what mattered for transfer learning. Ablation studies systematically remove or disable components of the model to measure each component's individual contribution. By comparing the full model against versions with specific components removed, researchers can distinguish which design choices contribute to performance from those that are present without affecting it. The GPT-1 ablations addressed three key questions: Does pre-training help? Does the auxiliary LM loss help? Does the transformer architecture help (compared to a simpler LSTM baseline)?