Inference Scaling › Limitations and Impact

Inference scaling analysis gives useful guidance but has useful limitations. The cost models presented here use simplified assumptions: they assume that throughput is constrained by either compute or memory bandwidth in isolation. Real systems exhibit more complex behavior where both constraints interact, speculative decoding changes the compute-per-token relationship, and quantization techniques (which we'll cover in later chapters) can materially shift efficiency.

The "tokens generated" framing also obscures the distinction between the prefill phase (processing the input prompt) and the decode phase (generating output tokens). Prefill is compute-bound and can be batched efficiently, while decode is typically memory-bound. Models serving long prompts with short outputs have different cost profiles than those generating lengthy responses from brief prompts.

Despite these limitations, inference scaling has influenced how the field develops and deploys models. The LLaMA model family demonstrated that inference-optimized training could produce models competitive with much larger alternatives. This work catalyzed the open-source LLM ecosystem. Organizations now routinely train smaller models on more data than Chinchilla would suggest, accepting higher training costs to reach better deployment economics.

This shift has also influenced architecture design. Techniques like grouped-query attention, which we covered in the LLaMA components chapter, reduce memory bandwidth requirements and improve inference efficiency. The focus on inference has accelerated research into quantization, speculative decoding, and other techniques that reduce the effective cost per token.