Compute-Optimal Training › Training Efficiency in Practice

Compute-optimal formulas assume you extract maximum value from each FLOP. In practice, many factors affect training efficiency, and optimizing these factors can significantly impact your effective compute budget. The gap between theoretical compute (what the formulas predict) and practical compute (what you achieve) can be substantial. Understanding this gap is essential for realistic planning and for identifying opportunities to improve training efficiency.

Model FLOPs Utilization (MFU) measures what fraction of theoretical peak hardware performance you achieve. This metric captures the cumulative effect of all inefficiencies in a training system, from memory bandwidth bottlenecks to communication overhead to operations that cannot fully utilize specialized hardware:

$$\text{MFU} = \frac{\text{Observed throughput (FLOPS)}}{\text{Peak hardware FLOPS}}$$

where:

• $\text{MFU}$: Model FLOPs Utilization, a ratio between 0 and 1 (often expressed as a percentage)
• $\text{Observed throughput}$: the actual floating-point operations per second achieved during training
• $\text{Peak hardware FLOPS}$: the theoretical maximum operations per second the hardware can perform

For example, an A100 GPU has a peak of 312 TFLOPS for FP16 tensor operations. If your training achieves 140 TFLOPS of actual compute, your MFU is $140/312 \approx 45\%$. The gap comes from memory bandwidth limitations, communication overhead, and operations that cannot fully utilize tensor cores. This 55% gap might seem wasteful, but it reflects real constraints. Moving data between GPU memory and compute units takes time; synchronizing gradients across GPUs requires communication; and not every operation in a transformer maps efficiently onto tensor cores.

Typical MFU values for LLM training range from 30% to 60%, depending on:

• Hardware configuration: Multi-GPU communication overhead reduces MFU as cluster size grows
• Batch size: Larger batches amortize fixed costs but may require more memory
• Model architecture: Some operations (LayerNorm, activations) are memory-bound rather than compute-bound
• Precision: Mixed precision training (FP16/BF16 with FP32 accumulation) typically achieves higher MFU than FP32

Each of these factors represents an engineering challenge with its own optimization strategies. Hardware communication overhead can be reduced through careful placement of operations, overlapping communication with computation, and using efficient collective primitives. Batch size choices involve tradeoffs between memory pressure, gradient quality, and throughput. Architecture decisions affect the ratio of compute-bound to memory-bound operations, with implications for how efficiently the hardware can be utilized.

A 30% improvement in MFU (from 30% to 60%) cuts training time in half. This makes MFU optimization a critical concern for compute-optimal training. The practical implication is that engineering effort invested in improving MFU pays dividends equivalent to acquiring additional hardware. Improving MFU from 30% to 60% is equivalent to doubling your GPU count at no additional hardware cost.

Learning rate scheduling significantly affects training efficiency. The learning rate controls how aggressively the optimizer updates model parameters in response to gradient signals. Too high, and training becomes unstable as updates overshoot optimal parameter values. Too low, and training proceeds sluggishly, wasting compute on inefficient updates. The standard approach for LLM training uses:

1. Warmup: Linear increase from near-zero to peak learning rate
2. Decay: Cosine or linear decay to a minimum value

The warmup phase serves a critical stability function. At initialization, model parameters are random and gradients can be unreliable. Large learning rates applied to noisy gradients cause training to diverge. By starting with small learning rates and gradually increasing, warmup allows the model to enter a stable region before aggressive optimization begins. The decay phase addresses a different concern: as training progresses and the model approaches a good solution, large learning rates cause the optimizer to overshoot and oscillate around the minimum. Gradually reducing the learning rate allows fine-grained convergence.

The key parameters are:

• Peak learning rate: Scales inversely with model size; larger models use smaller learning rates
• Warmup steps: Typically 0.1% to 1% of total training steps
• Minimum learning rate: Usually 10% of peak, though some schedules decay to zero

Batch size affects both training dynamics and compute efficiency. Understanding batch size requires recognizing that gradient descent operates on estimated gradients. Each batch provides a noisy estimate of the true gradient, with larger batches providing more accurate estimates at the cost of more computation per update. The critical batch size is the point beyond which larger batches provide diminishing returns. Below this threshold, doubling batch size nearly halves training time. Above it, larger batches waste compute on redundant gradient information.

The diminishing returns arise from the statistical nature of gradient estimation. With small batches, each sample contributes significant independent information about the gradient direction. As batch size grows, samples begin to provide redundant information, they agree on the gradient direction, so averaging more of them improves the estimate only marginally. The critical batch size marks the transition between these regimes.

Empirically, the critical batch size scales with loss. As training progresses, the gradient signal becomes noisier relative to its magnitude, meaning that averaging over more samples (larger batches) provides diminishing returns. The relationship is:

$$B_{\text{crit}} \propto L^{-1}$$

where:

• $B_{\text{crit}}$: the critical batch size, beyond which larger batches provide diminishing returns
• $L^{-1}$: the inverse of the current loss value

This inverse relationship has an intuitive explanation: when loss is high (early in training), gradients are large and consistent across samples, so even small batches provide reliable gradient estimates. As loss decreases and the model approaches convergence, gradients become smaller and noisier, requiring more samples to average out the noise and get a reliable signal. This is why early in training (when loss is high), smaller batches are optimal. As the model improves and loss drops, it can benefit from larger batches.

Think of it this way: early in training, the model is making large, obvious mistakes that any sample can reveal. A small batch suffices to identify that "the model thinks 'the' should follow 'the', and that's wrong." Later in training, the model's errors are subtle, and any individual sample might push the gradient in a misleading direction due to noise. Larger batches average out these idiosyncratic signals to reveal the true underlying gradient.