GPT-2 › Model Sizes: A Family of Models
Language AI HandbookAll chapters GPT-2 › Model Sizes: A Family of ModelsOne section · Sections in this chapter2 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
GPT-2 wasn't a single model but a family of four. OpenAI trained models ranging from 117 million to 1.5 billion parameters, letting systematic study of how capabilities scale with size. This approach would become standard practice for subsequent large language models.
The decision to train four models of different sizes rather than one final model was itself scientifically significant. By holding the architecture constant and varying only size, the researchers could isolate the effect of scale from the effect of architectural choices. Every observed improvement in the larger models could be attributed directly to having more parameters and more capacity to memorize and generalize. This family-of-models approach became a template: GPT-3 would later present six sizes, and Meta's LLaMA family would publish multiple scales for the same reason.
Think of the four GPT-2 sizes as a staircase, with each step roughly doubling or tripling the parameter count while using the exact same architectural blueprint. GPT-2 Small is essentially GPT-1 replicated. GPT-2 Medium represents a meaningful first expansion. GPT-2 Large approaches what was then considered a very large model. GPT-2 XL, at 1.5 billion parameters, was the final destination that OpenAI initially withheld from public release, citing concerns about misuse potential.
GPT-2 Small matches GPT-1's size at 117M parameters with 12 layers. GPT-2 Medium doubles this to 345M parameters with 24 layers. GPT-2 Large reaches 762M with 36 layers. GPT-2 XL, the full model, contains 1.5B parameters across 48 layers.
Architecture Specifications
The architectural specifications for each variant are:
| Parameter | GPT-2 Small | GPT-2 Medium | GPT-2 Large | GPT-2 XL |
|---|---|---|---|---|
| Layers () | 12 | 24 | 36 | 48 |
| Hidden size () | 768 | 1024 | 1280 | 1600 |
| Attention heads () | 12 | 16 | 20 | 25 |
| Head dimension () | 64 | 64 | 64 | 64 |
| Feed-forward size | 3072 | 4096 | 5120 | 6400 |
| Vocabulary size | 50,257 | 50,257 | 50,257 | 50,257 |
| Context length | 1024 | 1024 | 1024 | 1024 |
| Parameters | ~117M | ~345M | ~762M | ~1.5B |
The key architectural parameters are:
- : number of transformer layers (blocks stacked sequentially)
- : hidden dimension, the size of token representations flowing through the network
- : number of attention heads operating in parallel within each layer
- : dimension per attention head, determining how much information each head can capture
One of the most important patterns in this table is the one that does not change: the head dimension stays fixed at 64 across all four variants. This is a deliberate engineering choice. As the model grows, the researchers did not make individual attention heads larger; they added more heads and more layers instead. Think of it like a company that grows not by making each employee a superhero but by hiring more people with the same specialized role. Each attention head in GPT-2 XL does the same kind of work as each head in GPT-2 Small, just as one of 25 specialists rather than one of 12.
The 4x feed-forward expansion ratio (hidden size multiplied by 4 equals feed-forward size) also remains constant across all variants. Context length doubled from GPT-1's 512 to 1024 tokens, letting the model to condition on longer passages of text before making each prediction. This doubled context is significant: it means the model can attend to roughly two pages of text when generating each word, compared to GPT-1's single page.
Parameter Distribution
Let's compute the parameter distribution to understand where model capacity resides:
def count_gpt2_parameters(
vocab_size: int = 50257,
hidden_size: int = 768,
num_layers: int = 12,
max_position: int = 1024,
) -> dict:
"""Count parameters in each component of GPT-2."""
intermediate_size = hidden_size * 4
params = {}
# Embedding layers (token and position)
params["token_embeddings"] = vocab_size * hidden_size
params["position_embeddings"] = max_position * hidden_size
# Per-layer parameters
# Self-attention: Q, K, V projections + output projection (all combined in c_attn, c_proj)
attention_params = (
3 * (hidden_size * hidden_size) + 3 * hidden_size
) # c_attn
attention_params += hidden_size * hidden_size + hidden_size # c_proj
# Feed-forward: two linear layers (c_fc, c_proj)
ff_params = hidden_size * intermediate_size + intermediate_size # c_fc
ff_params += intermediate_size * hidden_size + hidden_size # c_proj
# Layer norms (2 per layer)
layernorm_params = 4 * hidden_size
params["per_layer"] = attention_params + ff_params + layernorm_params
params["all_layers"] = num_layers * params["per_layer"]
# Final layer norm
params["final_layernorm"] = 2 * hidden_size
# Output projection (tied with token embeddings, so not counted separately)
params["output_projection"] = 0 # Weight tying
# Total
params["embeddings_total"] = (
params["token_embeddings"] + params["position_embeddings"]
)
params["total"] = (
params["embeddings_total"]
+ params["all_layers"]
+ params["final_layernorm"]
)
return paramsGPT-2 Small: Embeddings: 39,383,808 (31.6%) Transformer Layers: 85,054,464 (68.3%) Total: 124,439,808 GPT-2 Medium: Embeddings: 52,511,744 (14.8%) Transformer Layers: 302,309,376 (85.2%) Total: 354,823,168 GPT-2 Large: Embeddings: 65,639,680 (8.5%) Transformer Layers: 708,387,840 (91.5%) Total: 774,030,080 GPT-2 XL: Embeddings: 82,049,600 (5.3%) Transformer Layers: 1,475,558,400 (94.7%) Total: 1,557,611,200
The parameter breakdown reveals where model capacity resides. For GPT-2 Small, embeddings constitute about 31% of parameters, but this fraction shrinks dramatically as models scale. GPT-2 XL dedicates over 90% of its parameters to transformer layers, meaning the large majority of learned knowledge resides in attention and feed-forward weights rather than the vocabulary embeddings.
The embedding layer represents a decreasing fraction of total parameters as models grow. In GPT-2 Small, embeddings account for roughly 31% of parameters, but in GPT-2 XL this drops to about 6%. This pattern reflects a basic principle: vocabulary embeddings scale with vocabulary size (50,257 tokens times hidden dimension), while transformer layers scale quadratically with hidden dimension and linearly with depth. For large models, knowledge increasingly resides in the transformer layers themselves, inside the weights of the attention projections and feed-forward networks.
The key insight here is that knowledge has a different home in small versus large models. In a small model, a disproportionate share of the network is just a lookup table mapping tokens to vectors. In a large model, the transformer machinery dominates, and that machinery is where the model stores relationships between concepts, syntactic patterns, and factual associations.