GPT-3 › The GPT-3 Model Family
Language AI HandbookAll chapters GPT-3 › The GPT-3 Model FamilyOne section · Sections in this chapter3 / 25 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
OpenAI released GPT-3 not as a single model but as a family of eight models spanning four orders of magnitude in size. This range enabled researchers to study how capabilities scale with model size and supplied the evidence for the paper's central claims.
The key insight behind releasing a model family rather than a single large model was methodological: if you claim that some capability emerges at scale, you need to show that smaller models do not have the capability. Without smaller models to compare against, you cannot distinguish between "this requires 175B parameters" and "this requires any reasonable language model." The systematic comparison across sizes transformed anecdotal observations about large models into rigorous empirical findings.
| Model Name | Parameters | Layers | Hidden Size | Heads |
|---|---|---|---|---|
| GPT-3 Small | 125M | 12 | 768 | 12 |
| GPT-3 Medium | 350M | 24 | 1024 | 16 |
| GPT-3 Large | 760M | 24 | 1536 | 16 |
| GPT-3 XL | 1.3B | 24 | 2048 | 24 |
| GPT-3 2.7B | 2.7B | 32 | 2560 | 32 |
| GPT-3 6.7B | 6.7B | 32 | 4096 | 32 |
| GPT-3 13B | 13B | 40 | 5120 | 40 |
| GPT-3 175B | 175B | 96 | 12288 | 96 |
The smallest GPT-3 model is roughly equivalent to GPT-1 in size, while the largest is over 1,000 times bigger. Each size variant has a distinct architecture with different hidden dimensions and head counts instead of being a truncated version of the 175B model. This design provides cleaner comparisons across compact architectures.
This range allowed researchers to study emergent capabilities: abilities that appear suddenly at certain scales rather than improving gradually. For many tasks, performance on the 125M and 350M models would be near chance level, then improve modestly up to 13B, and then jump dramatically at 175B. This non-linear behavior suggested that in-context learning is not a gradual improvement but a threshold phenomenon: something about the 175B-scale model enables a qualitatively different kind of adaptation.
An emergent capability is a behavior that appears in larger models but is absent or near-random in smaller ones, even when performance is measured on the same scale. The word "emergent" is somewhat contested: some researchers argue that apparent emergence is an artifact of metric choice (sharp thresholds in accuracy can obscure smooth improvements in log-probability), while others maintain that discrete phase transitions occur. GPT-3 supplied early evidence for emergence and made it a central debate in scaling research.
