Inference Scaling › Summary

This chapter explored how scaling laws change when inference costs dominate training costs, as they do for any widely-deployed model:

• The training-inference asymmetry means that models are trained once but may serve trillions of inference requests. When inference compute exceeds training compute, optimizing for training efficiency becomes counterproductive.

• Chinchilla is training-optimal, not deployment-optimal. The compute-optimal ratios minimize training cost to reach a given loss, but don't account for the per-token inference cost that accumulates over deployment.

• Over-training refers to training smaller models on more data than Chinchilla ratios suggest. This invests additional training compute to reach lower inference costs. This trade-off pays off rapidly for high-volume deployments.

• The break-even analysis determines when over-training becomes economical, specifically when the total inference savings exceed the additional training investment. For production systems, break-even often occurs at modest request volumes.

• Deployment cost modeling must account for hardware constraints, memory bandwidth limitations, energy costs, and the superlinear relationship between model size and inference cost.

• Practical model selection depends on expected inference volume, quality requirements, latency constraints, and infrastructure capabilities. The optimal choice balances these factors rather than optimizing any single metric.

The inference scaling perspective completes our picture of how to allocate compute across the model development lifecycle. Combined with the training-focused scaling laws from earlier chapters, you now have frameworks for optimizing both phases of the compute budget. The next chapter extends these ideas to predict how model performance varies across different capabilities and benchmarks.