Inference Scaling: Optimizing LLMs for Production Deployment

The scaling laws we explored in previous chapters, from Kaplan's initial discoveries to Chinchilla's compute-optimal ratios, share a common assumption: they optimize for training efficiency. Given a fixed compute budget, how do you allocate it between model size and training data to minimize loss? This framing makes sense from a research perspective, where training costs dominate and the goal is to push capabilities forward as quickly as possible.

But production deployments face a fundamentally different optimization problem. A model is trained once, then serves inference requests continuously, potentially billions of times over its operational lifetime. When inference compute dwarfs training compute by orders of magnitude, optimizing for training efficiency can be precisely the wrong strategy.

This chapter examines how to think about scaling when inference costs matter. We'll see why the compute-optimal models from Chinchilla are inefficient for deployment, explore the practice of "over-training" smaller models for inference efficiency, and develop frameworks for modeling total deployment costs. The key insight is that the optimal model for training and the optimal model for serving are often very different. Understanding this distinction is needed for practical LLM deployment.

Consider the lifecycle of a large language model. Training is an intensive, one-time process that might consume millions of GPU-hours over several weeks. Once complete, the model enters production, where it handles inference requests: generating text, answering questions, or processing documents. This serving phase can extend for months or years.

The asymmetry becomes clear when you examine actual compute consumption. Think about what happens each time you query a language model. Every single token the model generates requires a forward pass through the entire network. The computation activates billions of parameters through matrix multiplications. When a model serves millions of users, each generating hundreds or thousands of tokens per interaction, these individual operations add up to large compute requirements.

To quantify this, we can calculate the total FLOPs consumed during inference and compare it to training cost. A training run might use $C_{train} = 10^{24}$ FLOPs (roughly what it takes to train a 70B parameter model on 2 trillion tokens). If that model then serves 1 billion inference requests, each generating an average of 500 tokens, the total inference compute is:

where:

• $C_{inference}$: total compute required for all inference requests over the model's deployment
• $N_{requests}$: number of inference requests served
• $L_{output}$: average number of output tokens generated per request
• $\text{FLOPs per token}$: floating-point operations required to generate one token

This formula shows an important fact about deployment economics: inference cost grows linearly with each of these three factors. More users, longer responses, or larger models all increase the compute burden proportionally. The multiplicative structure means that even modest increases in any factor can sharply shift the overall compute balance.

For a forward pass through an $N$-parameter model, inference requires approximately $2N$ FLOPs per token (one multiply-add per parameter). This $2N$ figure is important because it reveals a key fact about transformer efficiency: every parameter in your model must be activated for every token you generate. There's no shortcut, no way to skip over unused portions of the network. The full weight matrix participates in every prediction. For our 70B model:

This is about 7% of training compute. This is significant, but training still dominates in this scenario. At this scale, the traditional wisdom of optimizing for training efficiency still holds. However, if the model serves 100 billion requests:

Now inference compute exceeds training compute by a factor of 7. This crossover point, where inference begins to dominate, determines whether training-optimal or inference-optimal strategies should guide model selection. Understanding where your deployment falls relative to this threshold is important for making economically sound decisions about model architecture and training strategy.

For the most widely deployed models serving trillions of requests, inference can consume 100× or more compute than training. Consider a model like GPT-4 or Claude serving millions of users worldwide, each engaging in multiple conversations per day over the course of years. The cumulative inference load dwarfs even the most expensive training runs.

This asymmetry fundamentally changes the optimization calculus. When training dominates, you want the most capable model you can train within your compute budget. When inference dominates, you want the smallest model that reaches acceptable quality, even if training it costs more per unit of capability gained. This shift in perspective is one of the most important practical insights in applied machine learning.

Recall from the Chinchilla scaling laws chapter that compute-optimal training allocates compute roughly equally between model size and data, following the relationship $D \approx 20N$ where $D$ is tokens trained and $N$ is parameters. This minimizes training FLOPs to reach a target loss. This ratio of roughly 20 tokens per parameter comes from empirical study and gives a clear recipe for efficient training.

But the Chinchilla framework optimizes for a specific objective: minimizing the compute required to reach the lowest possible loss during training. It treats training as the end goal rather than as a means to an end. When we recognize that the true objective is efficient deployment, not efficient training, the optimal strategy shifts sharply.

Consider two models trained to the same final loss:

Model A (Chinchilla-optimal):

• 70B parameters
• 1.4T tokens
• Training compute: $C_A = 6ND = 6 \times 70 \times 10^9 \times 1.4 \times 10^{12} \approx 5.9 \times 10^{23}$ FLOPs

The $6ND$ approximation comes from the fact that each training token requires a forward pass ($2N$ operations) plus a backward pass (approximately $4N$ operations for computing gradients), totaling $6N$ operations per token across $D$ tokens. This 6-to-1 ratio between training and inference compute per token explains why training is so much more expensive than inference on a per-token basis, but also why inference costs accumulate so quickly when serving many users.

where:

• $N$: number of model parameters
• $D$: number of training tokens
• $6ND$: total FLOPs, accounting for forward pass (2 operations per parameter per token) plus backward pass (roughly 4 operations per parameter per token)

Model B (Over-trained):

• 7B parameters
• 4T tokens (requires longer training to match Model A's loss)
• Training compute: $C_B = 6 \times 7 \times 10^9 \times 4 \times 10^{12} = 1.68 \times 10^{23}$ FLOPs

Model B uses less training compute to reach the same loss. How can this be? The key is that Chinchilla optimality assumes you're pushing to the frontier of what's achievable at a given compute level. If you're willing to accept a specific target loss rather than minimizing loss for fixed compute, smaller models trained longer can sometimes be more efficient even for training.

This counterintuitive result arises because Chinchilla optimality answers a different question than the one we're asking. Chinchilla asks: "Given a fixed compute budget, how do I get the lowest possible loss?" The deployment question asks: "Given a target quality level, how do I minimize total cost over the model's lifetime?" These are fundamentally different optimization problems with different optimal solutions.

The real advantage of smaller models emerges at inference time. Model A requires $2 \times 70 \times 10^9 = 1.4 \times 10^{11}$ FLOPs per output token, while Model B requires $2 \times 7 \times 10^9 = 1.4 \times 10^{10}$ FLOPs per token. This means 10× fewer inference FLOPs for equivalent quality.

This 10× reduction in per-token cost compounds with every inference request. If both models serve a billion requests, Model B saves approximately $1.26 \times 10^{20}$ FLOPs per token generated. This compute savings translates directly to reduced electricity bills, fewer GPUs required, and lower operational costs.

For high-volume deployments, this 10× inference efficiency can translate to large cost savings that dwarf any training cost difference.

"Over-training" refers to training a model on more tokens than Chinchilla optimality would suggest. The term is slightly misleading, as it's only "over" training relative to compute-optimal. It's exactly the right amount of training for inference-optimal deployment. Think of it not as excessive training but as investing additional training compute to reduce inference costs later.

The LLaMA model family explicitly adopted this philosophy. The LLaMA paper states: "We only consider the performance of the resulting model, not the training compute budget." They trained 7B, 13B, 33B, and 65B parameter models on up to 1.4 trillion tokens. This is far more than Chinchilla ratios would suggest for the smaller models.

This statement is a philosophical break from the Chinchilla paradigm. Rather than asking how to minimize training compute, Meta's researchers asked how to maximize the value delivered per inference FLOP. This reframing led to models that were "inefficient" to train but were much more efficient to deploy.

For the 7B model, Chinchilla optimality would suggest around 140B tokens. Training on 1.4T tokens is 10× "over-training." This extra training compute is an investment that pays dividends at inference time through reduced per-token cost. Every additional training token helps the smaller model close the capability gap with larger alternatives, which makes it suitable for tasks that would otherwise require a more expensive model.

We can quantify over-training with a simple ratio that captures how far a model departs from Chinchilla-optimal training:

where:

• $D_{actual}$: the number of tokens used to train the model
• $D_{optimal}$: the Chinchilla-optimal number of tokens for a model of size $N$
• $N$: the number of model parameters
• $20N$: the Chinchilla approximation that optimal training uses roughly 20 tokens per parameter

This ratio gives a simple metric for characterizing how inference-optimized a model's training regime is. A ratio of 1 means the model was trained according to Chinchilla-optimal principles. Ratios greater than 1 indicate over-training for inference efficiency, with higher values representing more aggressive investment in training compute to reduce inference costs.

The table below shows typical over-training ratios for inference-optimized models:

Modern inference-optimized models routinely use over-training ratios of 10× to 50× or more. The trend toward higher ratios shows growing recognition that inference costs dominate for production systems. Mistral's 57× over-training ratio on their 7B model shows how far this philosophy has been pushed. They invested roughly 57 times more training compute than Chinchilla would recommend. They bet that the resulting inference efficiency would more than compensate.

Over-training makes economic sense when total inference compute exceeds the extra training compute required. Let's formalize this break-even analysis.

Suppose you have two options:

• Option 1: Train a large model $N_1$ on $D_1$ tokens (Chinchilla-optimal)
• Option 2: Train a smaller model $N_2 < N_1$ on $D_2 > D_1$ tokens to match Option 1's loss

The additional training cost for Option 2 is:

where:

• $\Delta C_{train}$: the difference in training compute between the two options
• $N_1, N_2$: parameter counts for Option 1 (larger) and Option 2 (smaller) respectively
• $D_1, D_2$: training tokens for Option 1 and Option 2 respectively

This difference can be positive or negative depending on the specific model sizes and training durations. If the smaller model requires enough additional training to offset the reduced parameter count, $\Delta C_{train}$ will be positive, representing an upfront investment that must be recouped through inference savings.

The inference savings per token generated are:

Here, $\Delta C_{inference}$ is the FLOPs saved per output token by using the smaller model (since each forward pass requires $2N$ FLOPs per token).

This quantity is always positive when $N_1 > N_2$, meaning smaller models always save inference compute regardless of how they were trained. The magnitude of these savings grows linearly with the difference in model sizes. A 63B parameter reduction saves 63 billion fewer multiply-add operations per token.

Over-training pays off when total inference tokens $T_{inference}$ satisfies:

This break-even analysis determines whether the inference savings justify the training investment. The inequality has a clear interpretation: the left side is cumulative inference savings over the model's deployment, while the right side is the upfront training cost differential. When expected inference volume exceeds the break-even point, the smaller over-trained model becomes the economically rational choice.

Let's work through a concrete comparison between a 70B Chinchilla-optimal model and a 7B over-trained model, both achieving similar loss. This example illustrates the practical magnitude of the cost differences and helps build intuition for deployment decisions.

Option 1: 70B model, 1.4T tokens

• Training: $6 \times 70B \times 1.4T = 5.9 \times 10^{23}$ FLOPs
• Inference: $2 \times 70B = 1.4 \times 10^{11}$ FLOPs/token

Option 2: 7B model, 4T tokens (assume this matches 70B model's loss)

• Training: $6 \times 7B \times 4T = 1.68 \times 10^{23}$ FLOPs
• Inference: $2 \times 7B = 1.4 \times 10^{10}$ FLOPs/token

In this case, Option 2 costs less to train. This can happen when targeting a specific loss level rather than frontier performance. The inference savings are:

Every inference token saves $1.26 \times 10^{11}$ FLOPs, and there's no training penalty. This is a best-case scenario where the over-trained model wins on both training and inference efficiency. It highlights how the conventional wisdom about "larger models being better" can be misleading for deployment contexts.

Even when over-training does cost more, break-even typically occurs at modest inference volumes. If the 7B model required $10^{24}$ FLOPs to train (an additional $8 \times 10^{23}$ FLOPs over the 70B option), the break-even would be:

At 500 tokens per request, that's about 12.6 billion inference requests, easily exceeded by any widely-deployed model. To put this in perspective, a popular API serving just 10,000 users, each making 10 requests per day, would reach this break-even point in roughly 3.4 years. For consumer-scale deployments with millions of users, break-even occurs within days or weeks of launch.

Given expected inference demand, we can determine the optimal model size directly. This requires understanding how loss scales with both model size and training data, and then reformulating the optimization problem to account for total lifecycle costs rather than just training costs.

Building on the Chinchilla loss function from earlier chapters:

where:

• $L(N, D)$: the model's loss as a function of size and training data
• $N$: the number of model parameters
• $D$: the number of training tokens
• $A, B$: empirically fitted scaling coefficients
• $\alpha, \beta$: empirically fitted exponents (typically around 0.34 and 0.28)
• $E$: the irreducible error floor representing the entropy of natural language

This functional form captures two distinct pathways to reducing loss: adding more parameters (the first term) or training on more data (the second term). The irreducible error floor $E$ is the basic uncertainty in language. Even a perfect model cannot predict with certainty what word comes next in human text. Understanding this structure is needed for reasoning about the trade-offs between model size and training duration.

We can reformulate the optimization problem for inference-dominant scenarios. Rather than minimizing loss for a fixed training budget (the Chinchilla approach), we want to minimize loss for a fixed total budget that includes both training and expected inference. This reformulation shifts the optimization target from training efficiency to deployment efficiency.

The total compute over the model's lifecycle is:

where:

• $C_{total}$: total compute over the model's entire lifecycle
• $C_{train} = 6ND$: training compute (approximately 6 FLOPs per parameter per token)
• $C_{inference} = 2N \cdot T_{inference}$: total inference compute (2 FLOPs per parameter per token generated)
• $T_{inference}$: expected total tokens generated during the model's deployment

This formulation captures the key insight: training cost is paid once, but inference cost accumulates with every token generated. The training term depends on both $N$ and $D$ (model size times data), while the inference term depends on $N$ and $T_{inference}$ (model size times usage). This asymmetric structure drives the preference for smaller models in high-inference scenarios.

To minimize loss subject to a total compute budget, we take derivatives and solve. The resulting optimal allocation shifts toward smaller models as expected inference volume increases:

where:

• $N^*$: the optimal model size (number of parameters) for the given deployment scenario
• $C_{total}$: the total compute budget available across training and inference
• $T_{inference}$: expected total tokens to be generated during deployment
• $a, b$: positive exponents derived from the scaling law parameters $\alpha$ and $\beta$. The exact values depend on the specific scaling law coefficients, but typical values yield $a \approx 0.5$ and $b \approx 0.3$

The negative exponent on $T_{inference}$ is the key result: it shows mathematically that optimal model size decreases as expected inference volume increases. This relationship emerges directly from the calculus of constrained optimization. When inference tokens grow, the cost of each parameter (which must be activated for every token) grows proportionally, pushing the optimum toward fewer parameters.

The intuition is clear: as inference demand grows, the optimal model size shrinks because each parameter incurs a cost at every inference, while training cost is amortized. A parameter that costs billions of FLOPs to train well might seem expensive, but if that parameter is then used trillions of times at inference, the training cost becomes negligible compared to the cumulative inference burden.

For deployment planning, a simplified heuristic helps guide model selection. If you expect to generate $T$ inference tokens over the model's lifetime, compare the inference compute to training compute:

where:

• $T$: total inference tokens expected over the model's deployment lifetime
• $D$: number of tokens used for training
• $N$: model parameters (which cancel out in the simplification)

Notice that $N$ cancels out. The inference factor depends only on the ratio of inference tokens to training tokens. This cancellation is mathematically elegant and practically useful because you can evaluate whether inference will dominate without even knowing the model size. When this factor exceeds 1, inference compute dominates. The larger this factor, the more aggressively you should favor smaller, over-trained models.

The inference factor gives a simple decision rule. Values below 1 indicate training-dominated scenarios where Chinchilla-optimal sizing remains appropriate. Values above 1 signal that inference costs will outweigh training costs, justifying investment in smaller, more heavily trained models. Values above 10 suggest aggressive over-training is warranted. Values above 100 indicate that inference efficiency should be the primary design consideration.

Real deployment decisions involve more than FLOP counting. Hardware costs, memory constraints, latency requirements, and energy consumption all factor into the total cost of ownership. Moving from theoretical compute analysis to practical cost modeling requires understanding how FLOPs translate to dollars and how hardware constraints shape achievable throughput.

A complete cost model includes:

• Hardware amortization: The cost of training and inference hardware amortized over useful life
• Compute costs: Electricity and cooling, proportional to FLOPs executed
• Memory costs: Larger models require more expensive hardware configurations
• Latency penalties: Slower inference may reduce user engagement or transaction value
• Opportunity costs: Resources committed to one model can't serve others

For a concrete cost analysis, we model the cost to generate one inference token. This simple economic model divides fixed hourly costs by throughput: fewer tokens per hour means each token must bear a larger share of the infrastructure cost:

This formula reveals the basic economic structure of inference: you're paying for GPU time, and the question is how many tokens you can squeeze out of each hour of that expensive hardware. Higher throughput means lower per-token costs, which is why inference optimization focuses so heavily on maximizing tokens per second.

Tokens per hour depends on model throughput, which is constrained by either compute or memory bandwidth:

where:

• $\text{Throughput}$: tokens generated per second
• $\text{FLOPS capacity}$: the GPU's floating-point operations per second
• $2N$: FLOPs required per token (two operations per parameter for the forward pass)
• $\text{Memory bandwidth}$: bytes per second the GPU can read from memory
• $\text{bytes per param}$: memory footprint per parameter (e.g., 2 bytes for FP16)

The minimum captures a basic hardware constraint: generation speed is limited by whichever resource (compute or memory bandwidth) is exhausted first. This bottleneck analysis is important for understanding real-world inference performance.

The first term ($\text{FLOPS capacity}/2N$) is the theoretical maximum if compute were the only constraint, dividing total available operations by operations needed per token. This would be the limit if weights could be loaded from memory instantaneously.

The second term captures the memory bottleneck: each token generation requires loading all model weights from memory, so throughput cannot exceed memory bandwidth divided by model size. The autoregressive generation process must stream the entire weight matrix through the GPU's memory bus for every single token produced.

For large models, memory bandwidth typically dominates, making smaller models even more advantageous than pure FLOP analysis suggests. Modern GPUs have tremendous compute capacity but relatively limited memory bandwidth. An A100 can perform 312 trillion floating-point operations per second but can only move 2 trillion bytes per second from memory. For large models, the weights cannot be loaded fast enough to keep the compute units busy.

The following code implements a deployment cost model incorporating these factors:

For capacity planning, we can project monthly costs based on expected request volume:

Inference scaling decisions involve more than raw compute costs. Several practical factors influence the optimal model size.

Over-training has diminishing returns. As you train a smaller model on more data, it eventually hits capacity limits, and additional training gives minimal benefit. The loss curve for a fixed model size approaches an asymptote:

This equation shows the basic capacity limit of any fixed-size model: no amount of additional training data can push the loss below the floor set by model size and the irreducible entropy of language.

where:

• $L(N, D)$: the model's loss as a function of parameters $N$ and training data $D$
• $A$: empirically fitted scaling coefficient for model capacity
• $N^\alpha$: model size raised to the scaling exponent (typically $\alpha \approx 0.34$)
• $E$: irreducible error floor (entropy of natural language)
• $D \to \infty$: the limit as training data approaches infinity

This limit shows what happens when a fixed-size model is trained on infinite data: the data-dependent term $B/D^\beta$ vanishes, leaving only the model capacity term $A/N^\alpha$ and the irreducible error floor $E$. The model capacity term is limitations inherent to the architecture. A 7B model simply cannot stand for certain complex patterns that a 70B model can capture. These architectural constraints set hard limits on how much over-training can compensate for fewer parameters.

Research suggests that models can be productively over-trained by roughly 10-50× before returns diminish materially. Beyond that, you're paying for training compute that yields minimal quality improvement.

Different tasks have different sensitivity to model size. Smaller models may:

• Struggle with complex reasoning chains
• Have weaker few-shot learning compared to larger models
• Show degraded performance on rare or specialized domains
• Handle adversarial or unusual inputs less reliably

For applications requiring these capabilities, accepting higher inference costs may be necessary. The next chapter on predicting model performance will give frameworks for estimating these task-specific capabilities.

Larger models require proportionally more GPU memory. A 70B FP16 model needs 140GB just for weights, exceeding single-GPU capacity and requiring model parallelism. This introduces:

• Higher latency from cross-GPU communication
• Reduced batch efficiency
• More complex deployment infrastructure

Smaller models that fit on a single GPU often reach better effective throughput despite needing to load parameters for every token.

Production inference servers batch multiple requests together, amortizing the cost of loading model weights across many tokens. Batching efficiency improves with model size to a point, but larger models have smaller maximum batch sizes due to memory constraints.

The optimal batch size trades off:

• Throughput (higher batches = better GPU utilization)
• Latency (larger batches = longer queue times)
• Memory (each request consumes activation memory)

The key parameters for inference-optimal deployment are:

• Over-training ratio: The factor by which actual training tokens exceed Chinchilla-optimal tokens ($D_{actual}/20N$). Ratios of 10-50× are common for inference-optimized models.
• Inference factor: The ratio of expected inference tokens to training tokens ($T/3D$). When this exceeds 1, inference costs dominate and smaller models become preferable.
• Memory bandwidth: Often the limiting factor for large model inference throughput, not raw compute capacity.
• Bytes per parameter: Determines model memory footprint (2 bytes for FP16, 1 byte for INT8). Smaller values let larger batch sizes and better throughput.

Inference scaling analysis gives useful guidance but has useful limitations. The cost models presented here use simplified assumptions: they assume that throughput is constrained by either compute or memory bandwidth in isolation. Real systems exhibit more complex behavior where both constraints interact, speculative decoding changes the compute-per-token relationship, and quantization techniques (which we'll cover in later chapters) can materially shift efficiency.

The "tokens generated" framing also obscures the distinction between the prefill phase (processing the input prompt) and the decode phase (generating output tokens). Prefill is compute-bound and can be batched efficiently, while decode is typically memory-bound. Models serving long prompts with short outputs have different cost profiles than those generating lengthy responses from brief prompts.

Despite these limitations, inference scaling has influenced how the field develops and deploys models. The LLaMA model family demonstrated that inference-optimized training could produce models competitive with much larger alternatives. This work catalyzed the open-source LLM ecosystem. Organizations now routinely train smaller models on more data than Chinchilla would suggest, accepting higher training costs to reach better deployment economics.

This shift has also influenced architecture design. Techniques like grouped-query attention, which we covered in the LLaMA components chapter, reduce memory bandwidth requirements and improve inference efficiency. The focus on inference has accelerated research into quantization, speculative decoding, and other techniques that reduce the effective cost per token.

This chapter explored how scaling laws change when inference costs dominate training costs, as they do for any widely-deployed model:

• The training-inference asymmetry means that models are trained once but may serve trillions of inference requests. When inference compute exceeds training compute, optimizing for training efficiency becomes counterproductive.

• Chinchilla is training-optimal, not deployment-optimal. The compute-optimal ratios minimize training cost to reach a given loss, but don't account for the per-token inference cost that accumulates over deployment.

• Over-training refers to training smaller models on more data than Chinchilla ratios suggest. This invests additional training compute to reach lower inference costs. This trade-off pays off rapidly for high-volume deployments.

• The break-even analysis determines when over-training becomes economical, specifically when the total inference savings exceed the additional training investment. For production systems, break-even often occurs at modest request volumes.

• Deployment cost modeling must account for hardware constraints, memory bandwidth limitations, energy costs, and the superlinear relationship between model size and inference cost.

• Practical model selection depends on expected inference volume, quality requirements, latency constraints, and infrastructure capabilities. The optimal choice balances these factors rather than optimizing any single metric.

The inference scaling perspective completes our picture of how to allocate compute across the model development lifecycle. Combined with the training-focused scaling laws from earlier chapters, you now have frameworks for optimizing both phases of the compute budget. The next chapter extends these ideas to predict how model performance varies across different capabilities and benchmarks.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about inference scaling and deployment optimization.