Data Parallelism › Scaling Behaviour

Understanding how DDP scales is essential for planning distributed training runs. Ideal scaling and real-world scaling differ in important ways, and knowing these differences prevents unpleasant surprises when you provision hardware.

In the ideal case, adding $N$ GPUs gives you an $N$-fold speedup. With batch size $b$ per GPU, each training step processes $N \times b$ total samples. Since one step on $N$ GPUs takes the same wall-clock time as one step on 1 GPU (assuming communication is free), you process $N$ times as many samples per unit time. Training runs $N$ times faster.

This ideal scaling holds when:

1. Communication time is negligible compared to compute time
2. The workload divides evenly among GPUs (no load imbalance)
3. All GPUs complete their local computation at the same time (no stragglers)
4. Data loading keeps up with GPU throughput

For large models with many parameters, condition 1 often approximately holds because compute time scales with model size while communication volume also scales with model size but at a lower constant factor (ring all-reduce transfers only $2|\mathbf{g}|$ bytes per rank regardless of $N$). A 7B model has 14 GB of BF16 gradients but requires several hundred milliseconds of compute per step, so the communication-to-compute ratio can be small on fast interconnects.

In practice, some fraction of each training step is inherently serial or requires synchronization. Amdahl's Law captures this fundamental limit: if fraction $p$ of the work can be parallelized and fraction $1-p$ is serial, the maximum speedup from $N$ processors is:

$$S(N) = \frac{1}{(1-p) + \frac{p}{N}}$$

where:

• $S(N)$ is the speedup with $N$ GPUs relative to 1 GPU
• $p$ is the fraction of total time that scales with more GPUs (the parallelizable part)
• $1-p$ is the serial fraction (overhead from synchronization, data loading, communication, and other fixed costs)

As $N \to \infty$, the speedup approaches $\frac{1}{1-p}$. Even with a serial fraction of just 5%, the maximum possible speedup is 20x regardless of how many GPUs you add. At 10% serial fraction, the ceiling drops to 10x. These limits are strict: no amount of engineering can overcome them without reducing the serial fraction.

For DDP, the main non-parallelizable overhead is the all-reduce communication that cannot be fully overlapped with compute. The communication time for ring all-reduce is approximately:

$$t_{\text{comm}} \approx 2 \cdot \frac{N-1}{N} \cdot \frac{|\mathbf{g}|}{B_{\text{link}}}$$

where:

• $|\mathbf{g}|$ is the gradient tensor size in bytes
• $B_{\text{link}}$ is the available link bandwidth per GPU in bytes per second (this is bidirectional bandwidth; NCCL can use both send and receive channels simultaneously)
• The factor approaches 2 as $N$ grows large (both reduce-scatter and all-gather phases each transfer approximately $|\mathbf{g}|$ bytes per rank)

This communication time is independent of $N$ for large $N$, because ring all-reduce is bandwidth-optimal. The implication is that DDP scales very well within a single node (NVLink bandwidth is high, so communication is fast) but scales less well across nodes (InfiniBand bandwidth is lower). Each additional node adds another cross-node communication hop, and if the cross-node bandwidth is already saturated, scaling efficiency degrades.

When scaling DDP to more GPUs, you should also scale the global learning rate. The standard practice, established by Goyal et al. (2017) in their ImageNet training paper and confirmed for language models by many subsequent works, is the linear scaling rule:

$$\alpha_N = \alpha_1 \times N$$

where:

• $\alpha_1$ is the learning rate for 1-GPU training with local batch size $b$
• $\alpha_N$ is the learning rate for $N$-GPU training with global batch size $Nb$

The intuition is this: with $N$ times as many samples per update step, each gradient estimate is $N$ times less noisy (by the central limit theorem, the variance of a mean over $N$ samples is $1/N$ times the variance of a single sample). Lower gradient noise means each update step is more reliable, so you can afford to take $N$ times larger steps. Taking larger steps means you cover the same loss landscape in fewer steps, maintaining equivalent training dynamics.

The linear scaling rule works well for batch sizes up to a few thousand examples. For very large batch sizes (beyond 8,000 to 16,000 for typical language model training), learning rate scaling alone is insufficient. At these scales, the gradient noise is so low that each update consumes nearly all of the useful curvature information available in the loss landscape in a single step, and you enter the "large batch regime" where additional batch size no longer reduces the number of steps needed proportionally. Research by McCandlish et al. (2018) on the "critical batch size" formalized this regime.

Learning rate warmup is also essential when scaling to large batch sizes. Rather than jumping immediately to the scaled learning rate $\alpha_N$, you linearly increase the learning rate from a small initial value (or zero) to $\alpha_N$ over the first few thousand training steps. This warmup period prevents instability at the start of training when the optimizer's first and second moment estimates (in Adam/AdamW) have not yet converged and the model is far from any sensible initialization.