Inference Scaling › Key Parameters

The key parameters for inference-optimal deployment are:

• Over-training ratio: The factor by which actual training tokens exceed Chinchilla-optimal tokens ($D_{actual}/20N$). Ratios of 10-50× are common for inference-optimized models.
• Inference factor: The ratio of expected inference tokens to training tokens ($T/3D$). When this exceeds 1, inference costs dominate and smaller models become preferable.
• Memory bandwidth: Often the limiting factor for large model inference throughput, not raw compute capacity.
• Bytes per parameter: Determines model memory footprint (2 bytes for FP16, 1 byte for INT8). Smaller values let larger batch sizes and better throughput.