Scaling Frontiers › Economics and Access: Who Can Scale?
Language AI HandbookAll chapters Scaling Frontiers › Economics and Access: Who Can Scale?One section · Sections in this chapter7 / 10 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The economics of frontier AI scaling shape the pace of progress and its distribution. As training costs rise, the number of organizations capable of frontier training shrinks. This concentration has several effects.
Centralization Risk
When only a handful of organizations can train frontier models, the diversity of approaches and objectives in the field narrows. Different organizations may have different priorities: some optimize for commercial deployment, others for scientific research, others for safety. A more distributed research ecosystem produces more diverse experiments and more reliable error correction when individual approaches fail.
The current trajectory points toward further centralization. The compute costs of training frontier models in 2024 are already beyond the reach of most universities, most startups, and most governments outside a small number of wealthy nations. If this trend continues, future models will be created by an increasingly small group of actors.
Open-source models and efficient fine-tuning techniques have partially offset this concern. Meta's LLaMA releases made high-quality pretrained weights publicly available, enabling a large ecosystem of researchers to build on frontier-quality foundations without reproducing the training costs. PEFT techniques like LoRA, which we covered in Part N, reduce the compute required for adaptation to specific tasks.
But open-source access to pre-trained weights is not equivalent to the ability to train new frontier models. An organization that can only fine-tune existing models is constrained to the capabilities of whoever produced those base models. The base model determines what the fine-tuned model can and cannot do. If frontier base models are produced by a very small number of actors, then even a vibrant open-source ecosystem is ultimately dependent on those actors.
Inference Economics and Model Distillation
Even without addressing training costs, the inference cost of running frontier models creates access barriers. A large language model inference call requires significant compute, and at scale this translates to substantial costs per query. Many valuable applications, particularly in low-income markets and public sector use cases, are not economically viable at current pricing.
Model distillation addresses this by training smaller, cheaper models to imitate the behavior of larger ones. We covered distillation in Part Q on model compression. The key insight relevant here is that a smaller distilled model, trained on the outputs of a larger teacher model, can often achieve a substantial fraction of the teacher's performance at a fraction of the inference cost.
Distillation is increasingly important as a compression technique and as a democratization mechanism. If a research organization trains a frontier model and then distills it into a much smaller model that is released publicly, the resulting smaller model may be affordable for a much wider range of applications. The quality of this smaller model depends critically on what it was distilled from, which brings us back to the importance of the frontier training.
The interaction between training economics, inference economics, and access creates a complex system. Progress in efficient architectures that lower inference costs and distillation techniques that transfer capability to smaller models can partially offset the centralizing effects of high training costs.
import numpy as np
# Model training and inference cost scaling
# Training costs scale roughly with parameter count and data
# Inference costs scale with parameter count per token
def training_cost_millions(
params_billions, tokens_trillions, gpu_hour_cost=2.0
):
"""
Rough training cost estimate.
Chinchilla-optimal compute C ≈ 6 * N * D FLOPs.
GPU efficiency: ~300 TFLOPS per A100, ~3600 GPU-hours per exaFLOP.
"""
flops = 6 * (params_billions * 1e9) * (tokens_trillions * 1e12)
exaflops = flops / 1e18
gpu_hours = exaflops * 3600
cost = gpu_hours * gpu_hour_cost / 1e6 # in millions USD
return cost
def inference_cost_per_million_tokens(params_billions, gpu_hour_cost=2.0):
"""
Rough inference cost per million tokens generated.
Approximate: 2 FLOPs per parameter per token.
"""
flops_per_token = 2 * params_billions * 1e9
flops_per_million = flops_per_token * 1e6
exaflops = flops_per_million / 1e18
gpu_hours = exaflops * 3600
return gpu_hours * gpu_hour_cost
model_sizes = np.array([1, 7, 13, 34, 70, 175, 500, 1000]) # billions of params
# Chinchilla-optimal tokens: roughly 20x parameter count
optimal_tokens = model_sizes * 20 / 1e3 # in trillions
training_costs = np.array(
[training_cost_millions(n, d) for n, d in zip(model_sizes, optimal_tokens)]
)
inference_costs = np.array(
[inference_cost_per_million_tokens(n) for n in model_sizes]
)
Training costs grow superlinearly with model size because Chinchilla-optimal training requires proportionally more data as models scale. Inference costs grow linearly with parameters, creating sustained economic pressure toward smaller, more efficient models.