GPT-3 › The Scale of GPT-3

GPT-3's defining characteristic is its size. At 175 billion parameters, it dwarfed all previous language models. To understand what this means in practice, it helps to compare it with its predecessors and to think carefully about what parameters represent.

Each parameter is a single floating-point number, a weight in a matrix that the model adjusts during training to minimize its prediction error. Parameters encode the model's learned knowledge about language: grammar, facts, reasoning patterns, stylistic conventions, and conceptual associations. More parameters mean more representational capacity, which in principle allows the model to store and manipulate more complex patterns. The question GPT-3's creators asked was whether simply scaling up the number of parameters, together with proportionally more data and compute, would yield qualitatively new capabilities rather than just marginal improvements.

The answer, as the table below shows, involved a dramatic escalation across every architectural dimension:

The jump from GPT-2 to GPT-3 represents more than a linear improvement. With 96 transformer layers, a hidden dimension of 12,288, and 96 attention heads, GPT-3 can represent far more complex patterns and relationships. The context length of 2,048 tokens means the model can consider roughly 1,500 words of context when making predictions. Each of these numbers reflects a deliberate design choice about where to invest capacity. The hidden dimension controls how rich the representation of each token can be. The number of layers controls how many successive transformations the model applies before producing its output. The number of attention heads determines how many different "perspectives" the model can use when looking at relationships between tokens. GPT-3 pushed all of these dimensions far beyond what had been tried before.

Think of the hidden dimension as the vocabulary of internal concepts the model works with. GPT-1's 768-dimensional hidden state can represent 768 independent directions of variation in token meaning. GPT-3's 12,288-dimensional space offers 16 times as many dimensions, allowing much finer distinctions. A word like "bank" can be represented differently depending on whether the surrounding context involves rivers, money, or aviation, and a higher-dimensional space gives the model more room to encode these distinctions precisely.

The compute required for training grew even faster than the parameter count. This wasn't just about having more parameters; each parameter needed more training data to be utilized effectively. A rough estimate of training compute for a language model is:

$$C \approx 6 \cdot N \cdot D$$

where:

• $C$: total floating-point operations (FLOPs) for training
• $N$: number of model parameters
• $D$: number of training tokens
• The factor of 6 accounts for the forward and backward passes (approximately 2 FLOPs per parameter per token for the forward pass, and 4 for the backward pass, since the backward pass computes gradients with respect to both activations and weights)

For GPT-3 with $N = 175 \times 10^9$ parameters trained on $D \approx 300 \times 10^9$ tokens, this yields $C \approx 3.15 \times 10^{23}$ FLOPs, matching the reported training compute. This observation, later formalized in scaling laws, showed that optimal training balances model size with data quantity and training compute. You cannot simply make a model very large and train it briefly; the performance improvements from size only materialize when the model sees enough data to fill its representational capacity.