Inference Scaling › The Training-Inference Asymmetry

Consider the lifecycle of a large language model. Training is an intensive, one-time process that might consume millions of GPU-hours over several weeks. Once complete, the model enters production, where it handles inference requests: generating text, answering questions, or processing documents. This serving phase can extend for months or years.

The asymmetry becomes clear when you examine actual compute consumption. Think about what happens each time you query a language model. Every single token the model generates requires a forward pass through the entire network. The computation activates billions of parameters through matrix multiplications. When a model serves millions of users, each generating hundreds or thousands of tokens per interaction, these individual operations add up to large compute requirements.

To quantify this, we can calculate the total FLOPs consumed during inference and compare it to training cost. A training run might use $C_{train} = 10^{24}$ FLOPs (roughly what it takes to train a 70B parameter model on 2 trillion tokens). If that model then serves 1 billion inference requests, each generating an average of 500 tokens, the total inference compute is:

$$C_{inference} = N_{requests} \times L_{output} \times \text{FLOPs\_per\_token}$$

where:

• $C_{inference}$: total compute required for all inference requests over the model's deployment
• $N_{requests}$: number of inference requests served
• $L_{output}$: average number of output tokens generated per request
• $\text{FLOPs per token}$: floating-point operations required to generate one token

This formula shows an important fact about deployment economics: inference cost grows linearly with each of these three factors. More users, longer responses, or larger models all increase the compute burden proportionally. The multiplicative structure means that even modest increases in any factor can sharply shift the overall compute balance.

For a forward pass through an $N$-parameter model, inference requires approximately $2N$ FLOPs per token (one multiply-add per parameter). This $2N$ figure is important because it reveals a key fact about transformer efficiency: every parameter in your model must be activated for every token you generate. There's no shortcut, no way to skip over unused portions of the network. The full weight matrix participates in every prediction. For our 70B model:

$$C_{inference} = 10^9 \times 500 \times (2 \times 70 \times 10^9) = 7 \times 10^{22} \text{ FLOPs}$$

This is about 7% of training compute. This is significant, but training still dominates in this scenario. At this scale, the traditional wisdom of optimizing for training efficiency still holds. However, if the model serves 100 billion requests:

$$C_{inference} = 10^{11} \times 500 \times (2 \times 70 \times 10^9) = 7 \times 10^{24} \text{ FLOPs}$$

Now inference compute exceeds training compute by a factor of 7. This crossover point, where inference begins to dominate, determines whether training-optimal or inference-optimal strategies should guide model selection. Understanding where your deployment falls relative to this threshold is important for making economically sound decisions about model architecture and training strategy.

For the most widely deployed models serving trillions of requests, inference can consume 100× or more compute than training. Consider a model like GPT-4 or Claude serving millions of users worldwide, each engaging in multiple conversations per day over the course of years. The cumulative inference load dwarfs even the most expensive training runs.

This asymmetry fundamentally changes the optimization calculus. When training dominates, you want the most capable model you can train within your compute budget. When inference dominates, you want the smallest model that reaches acceptable quality, even if training it costs more per unit of capability gained. This shift in perspective is one of the most important practical insights in applied machine learning.