GPT-3 › Architectural Details

GPT-3 uses the same fundamental decoder-only transformer architecture as GPT-2. The core design remains unchanged: a stack of transformer decoder blocks with masked self-attention, each followed by a feed-forward network. This continuity was a deliberate choice. The GPT-3 authors wanted to isolate the effect of scale, not architecture innovation. By keeping the design consistent with GPT-2, any capability differences observed between the models can be attributed to size rather than architectural changes.

The key architectural differences from GPT-2 include:

• Alternating dense and sparse attention: In some layers, GPT-3 uses sparse attention patterns that attend to every other token, reducing computational cost for long sequences
• Learned position embeddings: Like GPT-2, positions are represented by learned embeddings rather than sinusoidal functions
• Pre-layer normalization: Layer normalization is applied before each sublayer (attention and FFN) rather than after, following the Pre-LN configuration that stabilizes training for deep networks

Pre-layer normalization (Pre-LN) deserves particular attention. In the original transformer design, layer normalization was applied after the residual connection, a pattern called Post-LN. For shallow networks, this works fine. But for very deep networks like GPT-3 with 96 layers, Post-LN creates gradient flow problems: the gradients must pass through each normalization operation during backpropagation, and at great depths this can cause gradients to vanish or explode. Pre-LN addresses this by normalizing before the sublayer operation. This keeps the residual connections carry gradients cleanly from output back to input regardless of depth. This seemingly minor change was essential for training stability at 96 layers.

When we say GPT-3 has 175 billion parameters, what exactly are we counting? Understanding where these parameters live reveals the model's structure and helps explain why scale matters so much for capability.

A transformer model stores its learned knowledge in weight matrices. Each weight is a single floating-point number that the model adjusts during training. The total count of these numbers determines the model's capacity to represent patterns in language. To understand GPT-3's scale, we need to trace through each component and count its parameters systematically.

The Building BlocksLink Copied

A GPT-style model consists of several distinct components, each contributing parameters:

1. Token embeddings: A lookup table that converts each vocabulary token into a dense vector
2. Position embeddings: A separate lookup table that encodes where each token appears in the sequence
3. Transformer layers: The repeated blocks that process and transform token representations
4. Final layer normalization: A normalization layer before the output projection
5. Output projection: A matrix that converts hidden states back to vocabulary probabilities

Let's formalize this. For a model with vocabulary size $V$, maximum sequence length $L$, model dimension $d$, number of layers $N$, and feed-forward dimension $d_{ff}$, the total parameter count is:

$$P_{\text{total}} = P_{\text{embed}} + P_{\text{pos}} + N \cdot P_{\text{layer}} + P_{\text{norm}} + P_{\text{output}}$$

where:

• $P_{\text{embed}} = V \cdot d$: token embedding parameters. Each of the $V$ vocabulary tokens gets its own $d$-dimensional vector, requiring $V \times d$ parameters total.
• $P_{\text{pos}} = L \cdot d$: position embedding parameters. Each of the $L$ possible positions gets a $d$-dimensional vector.
• $P_{\text{layer}}$: parameters per transformer layer, which we'll derive below.
• $P_{\text{norm}} = 2d$: final layer normalization has a learnable scale and bias vector, each of dimension $d$.
• $P_{\text{output}} = d \cdot V$: the output projection maps from hidden dimension $d$ back to vocabulary size $V$. This is often weight-tied with the token embeddings (sharing the same matrix), but GPT-3 uses separate weights.

The formula captures the intuition that larger vocabularies, longer sequences, and higher dimensions all increase parameter count. But the dominant term is $N \cdot P_{\text{layer}}$: the transformer layers, repeated $N$ times.

Inside a Transformer LayerLink Copied

Each transformer layer consists of two sublayers, each followed by layer normalization:

1. Multi-head self-attention: Allows tokens to gather information from other positions
2. Feed-forward network (FFN): Applies the same transformation to each position independently

The attention sublayer requires four projection matrices: Query ($\mathbf{W}_Q$), Key ($\mathbf{W}_K$), Value ($\mathbf{W}_V$), and Output ($\mathbf{W}_O$). Each matrix has shape $d \times d$, contributing $d^2$ parameters. With four matrices, attention adds $4d^2$ parameters.

The feed-forward network consists of two linear transformations. The first expands from dimension $d$ to $d_{ff}$ (typically $4d$), and the second contracts back from $d_{ff}$ to $d$. This requires $d \times d_{ff} + d_{ff} \times d = 2 \cdot d \cdot d_{ff}$ parameters.

Each layer also has two layer normalizations (one before attention, one before FFN in the Pre-LN configuration), each with scale and bias vectors of dimension $d$. This adds $4d$ parameters per layer.

Putting it together:

$$P_{\text{layer}} = P_{\text{attn}} + P_{\text{ffn}} + P_{\text{layer\_norms}}$$

where:

• $P_{\text{attn}} = 4d^2$: the four attention projection matrices ($\mathbf{W}_Q$, $\mathbf{W}_K$, $\mathbf{W}_V$, $\mathbf{W}_O$)
• $P_{\text{ffn}} = 2 \cdot d \cdot d_{ff}$: the two feed-forward linear layers. With $d_{ff} = 4d$, this becomes $8d^2$.
• $P_{\text{layer\_norms}} = 4d$: scale and bias for two layer normalizations

Notice that the per-layer count scales with $d^2$. This quadratic dependence explains why increasing the hidden dimension has such a dramatic effect on model size: doubling $d$ quadruples the parameters in each layer. It also explains why GPT-3's hidden dimension of 12,288 is so consequential: compared to GPT-2's 1,600, this is roughly an 8x increase in $d$, which translates to a 64x increase in parameters per layer from the quadratic terms alone.

A Worked Example: GPT-3 175BLink Copied

Let's apply these formulas to GPT-3's actual configuration:

• Vocabulary size: $V = 50257$ tokens (the GPT-2 tokenizer vocabulary)
• Maximum sequence length: $L = 2048$ tokens
• Model dimension: $d = 12288$
• Number of layers: $N = 96$
• Feed-forward dimension: $d_{ff} = 4 \times 12288 = 49152$

First, we calculate the per-layer parameters:

$$\begin{aligned} P_{\text{attn}} &= 4 \times 12288^2 = 4 \times 150{,}994{,}944 = 603{,}979{,}776 \\ P_{\text{ffn}} &= 2 \times 12288 \times 49152 = 1{,}207{,}959{,}552 \\ P_{\text{layer\_norms}} &= 4 \times 12288 = 49{,}152 \\ P_{\text{layer}} &= 603{,}979{,}776 + 1{,}207{,}959{,}552 + 49{,}152 \approx 1.81 \text{ billion} \end{aligned}$$

Each layer contributes about 1.81 billion parameters. With 96 layers, the transformer stack alone accounts for approximately 174 billion parameters. The remaining 1+ billion come from embeddings and the output projection.

This breakdown reveals an important pattern: the transformer layers dominate the parameter count. Increasing the number of layers or the hidden dimension has a much larger effect than expanding the vocabulary or context length. If you doubled the vocabulary to 100,514 tokens, the additional parameters would be less than 1.5 billion, barely 1% of the total. But if you doubled the hidden dimension to 24,576, you would quadruple the per-layer parameters and nearly quadruple the entire model. In practice, this means that the "knowledge" of a GPT-3-scale model lives overwhelmingly in its attention and FFN matrices, not in the embedding lookup tables.

ImplementationLink Copied

Let's implement this calculation to verify our formulas and explore how parameters distribute across components: