Inference Scaling › Trade-offs and Practical Considerations

Inference scaling decisions involve more than raw compute costs. Several practical factors influence the optimal model size.

Over-training has diminishing returns. As you train a smaller model on more data, it eventually hits capacity limits, and additional training gives minimal benefit. The loss curve for a fixed model size approaches an asymptote:

$$L(N, D) \to \frac{A}{N^\alpha} + E \quad \text{as } D \to \infty$$

This equation shows the basic capacity limit of any fixed-size model: no amount of additional training data can push the loss below the floor set by model size and the irreducible entropy of language.

where:

• $L(N, D)$: the model's loss as a function of parameters $N$ and training data $D$
• $A$: empirically fitted scaling coefficient for model capacity
• $N^\alpha$: model size raised to the scaling exponent (typically $\alpha \approx 0.34$)
• $E$: irreducible error floor (entropy of natural language)
• $D \to \infty$: the limit as training data approaches infinity

This limit shows what happens when a fixed-size model is trained on infinite data: the data-dependent term $B/D^\beta$ vanishes, leaving only the model capacity term $A/N^\alpha$ and the irreducible error floor $E$. The model capacity term is limitations inherent to the architecture. A 7B model simply cannot stand for certain complex patterns that a 70B model can capture. These architectural constraints set hard limits on how much over-training can compensate for fewer parameters.

Research suggests that models can be productively over-trained by roughly 10-50× before returns diminish materially. Beyond that, you're paying for training compute that yields minimal quality improvement.

Different tasks have different sensitivity to model size. Smaller models may:

• Struggle with complex reasoning chains
• Have weaker few-shot learning compared to larger models
• Show degraded performance on rare or specialized domains
• Handle adversarial or unusual inputs less reliably

For applications requiring these capabilities, accepting higher inference costs may be necessary. The next chapter on predicting model performance will give frameworks for estimating these task-specific capabilities.

Larger models require proportionally more GPU memory. A 70B FP16 model needs 140GB just for weights, exceeding single-GPU capacity and requiring model parallelism. This introduces:

• Higher latency from cross-GPU communication
• Reduced batch efficiency
• More complex deployment infrastructure

Smaller models that fit on a single GPU often reach better effective throughput despite needing to load parameters for every token.

Production inference servers batch multiple requests together, amortizing the cost of loading model weights across many tokens. Batching efficiency improves with model size to a point, but larger models have smaller maximum batch sizes due to memory constraints.

The optimal batch size trades off:

• Throughput (higher batches = better GPU utilization)
• Latency (larger batches = longer queue times)
• Memory (each request consumes activation memory)