Scaling Frontiers › The Limits of Scaling

The scaling paradigm rests on a set of empirical observations that have held surprisingly well. But empirical observations are not theorems. They describe what has happened, not what must happen. To understand where limits might emerge, we need to look at the physical, statistical, and economic constraints that bound the scaling regime.

The compute available for training grows through two channels: more efficient hardware and more hardware. Both have limits, though they operate on different timescales.

Hardware efficiency, historically captured by Moore's Law, has slowed dramatically. The density of transistors on chips can no longer double every 18 months as it once did. Physical constraints, particularly the atomic scale and heat dissipation, impose hard floors on how efficient individual chips can become. Specialized hardware like GPUs and TPUs has extended progress beyond what general-purpose CPUs could achieve, but even this avenue has narrowing returns.

The second channel is deploying more hardware in parallel. Modern training runs already use thousands of accelerators connected by specialized networking. The cost of frontier training runs reached hundreds of millions of dollars by 2024, with estimates for future runs reaching into the billions. This trajectory is unsustainable for all but a handful of organizations globally, and it raises questions about whether compute scaling can continue to be the primary driver of progress.

There is also a fundamental physical argument. Computation requires energy. Energy dissipation generates heat. Heat must be removed. At some scale, the thermal management requirements of training runs become comparable to industrial cooling operations. While this limit is far from what current technology reaches, it illustrates that scaling is bounded by physics, not just economics.

Beyond compute, scaling runs into a statistical limit: the irreducible entropy of natural language. As we discussed in the scaling laws chapters, loss functions of the form:

$$L(N) = \left(\frac{N_c}{N}\right)^{\alpha_N} + L_{\infty}$$

include a floor term $L_{\infty}$ representing the entropy of the data distribution itself. Once a model has learned everything learnable, further scaling cannot reduce loss below this floor.

where:

• $L(N)$: validation loss as a function of parameter count $N$
• $N_c$: a characteristic scale constant fitted to empirical data
• $\alpha_N$: the scaling exponent (typically around 0.076 for language models)
• $L_{\infty}$: the irreducible entropy floor representing inherent unpredictability in the data

This floor matters because it constrains benchmark performance. Many tasks that appear to test language understanding are testing the ability to predict text that follows recognizable patterns. As models approach the entropy floor on training data, the marginal improvement per additional parameter or token diminishes. The slope of the power law does not change, but you are squeezing a diminishing return from each resource unit.

The practical implication is that continued loss reduction requires either more training data, which faces the constraints we discuss next, or qualitatively different learning approaches. Scaling parameters alone cannot push below the floor.

Even where physical and statistical limits allow further scaling, economic reality may impose binding constraints. The cost-benefit analysis of scaling shifts as model size grows. Early in the scaling regime, each order of magnitude of compute delivered substantial benchmark improvements. If this relationship holds at smaller returns on larger investments, at some point the return on additional compute drops below the threshold where it is commercially viable.

Several factors accelerate this economic constraint. Energy costs are rising in many regions due to increased competition for power from AI, cryptocurrency, and general industrial demand. The supply chain for advanced semiconductor fabrication is concentrated in a small number of fabs, creating bottlenecks. And the labor required to curate, clean, and maintain training datasets grows superlinearly with dataset size.

These constraints do not make further scaling impossible. They mean that continued scaling requires solving hard engineering and logistical problems in parallel with the machine learning research, and that the organizations capable of doing so become fewer and more specialized.