Inference Scaling › Inference Scaling

The scaling laws we explored in previous chapters, from Kaplan's initial discoveries to Chinchilla's compute-optimal ratios, share a common assumption: they optimize for training efficiency. Given a fixed compute budget, how do you allocate it between model size and training data to minimize loss? This framing makes sense from a research perspective, where training costs dominate and the goal is to push capabilities forward as quickly as possible.

But production deployments face a fundamentally different optimization problem. A model is trained once, then serves inference requests continuously, potentially billions of times over its operational lifetime. When inference compute dwarfs training compute by orders of magnitude, optimizing for training efficiency can be precisely the wrong strategy.

This chapter examines how to think about scaling when inference costs matter. We'll see why the compute-optimal models from Chinchilla are inefficient for deployment, explore the practice of "over-training" smaller models for inference efficiency, and develop frameworks for modeling total deployment costs. The key insight is that the optimal model for training and the optimal model for serving are often very different. Understanding this distinction is needed for practical LLM deployment.