GPT-3 › The Role of the Pretraining Corpus in Shaping Capabilities

One of the most important and underappreciated aspects of GPT-3's success is how much the composition of its training corpus shaped its capabilities. The model did not develop its impressive zero-shot and few-shot abilities in a vacuum: it developed them because of exposure to specific kinds of text that implicitly taught the skills needed for in-context learning.

Consider the capability for arithmetic. GPT-3 was not trained on a math curriculum, but the internet contains an enormous amount of arithmetic-adjacent content: programming tutorials that compute intermediate values, science articles that perform calculations, educational websites that work through math problems step by step, and physics discussions that derive numerical answers. A model trained on this content is implicitly exposed to many demonstrations of arithmetic reasoning, encoded in natural language. It learns that numbers relate to each other in specific ways, that certain patterns of text (a series of numbers and operators) predict certain outputs (a result), and that arithmetic errors produce inconsistencies that would not appear in well-written text.

The same logic applies to translation, code generation, and commonsense reasoning. Each of these capabilities correlates with the presence of relevant text in the pretraining corpus. Languages that appear less frequently in the training corpus have weaker translation quality. Programming languages that appear less in training data have weaker code generation. Commonsense situations that are underrepresented in English web text are handled less reliably.

This dependence on corpus composition has a direct practical implication: the capabilities of a large language model are in some sense a mirror of its training data. A model trained predominantly on English text will be better at English tasks than tasks requiring other languages. A model trained predominantly on contemporary web content will reflect contemporary cultural knowledge and biases rather than historical ones. A model trained on filtered, high-quality text may have better factual accuracy but narrower coverage than one trained on raw web crawl data.

Understanding this relationship between training data and capabilities is essential for both using large language models effectively and for understanding their failure modes. When GPT-3 produces confidently wrong information about an obscure topic, the first question to ask is whether that topic is well-represented in its training data. When it handles some linguistic constructions better than others, the explanation often lies in the frequency of those constructions in its pretraining corpus. The model is, in a deep sense, a compression of its training data into a set of weights that can be queried through language.