Data Parallelism › Summary

Data parallelism is the foundational strategy for distributed language model training. By maintaining a complete model copy on each GPU and processing different data shards in parallel, DDP achieves near-linear speedup when communication bandwidth is sufficient relative to compute time.

The key ideas to carry forward are:

• DDP algorithm: Each rank computes local gradients on its data shard. All-reduce synchronizes gradients across all ranks. All ranks apply identical parameter updates. The model copies stay perfectly synchronized throughout training without any explicit parameter communication.
• All-reduce: Ring all-reduce is the bandwidth-optimal primitive that makes DDP scalable. Each rank transfers roughly $2|\mathbf{g}|$ bytes regardless of the number of GPUs, distributing aggregation work evenly across the ring. NCCL implements topology-aware variants that exploit NVLink for intranode communication and InfiniBand for cross-node traffic.
• Gradient bucketing: DDP overlaps communication with computation by launching all-reduce on completed gradient buckets during the backward pass, hiding most communication latency. The bucket size (default 25 MB) controls the tradeoff between communication launch overhead and overlap granularity.
• Scaling limits: Larger models scale more efficiently because their high compute-to-communication ratio keeps GPUs busy. Cross-node training with InfiniBand scales worse than within-node NVLink due to roughly 24x bandwidth difference. The linear scaling rule adjusts learning rates for larger global batch sizes, but above the critical batch size, total token cost increases.
• Memory cost: Pure data parallelism replicates the full model on every GPU, requiring 16 bytes per parameter for AdamW mixed precision training. For models exceeding GPU memory (above roughly 5B parameters on an A100), FSDP, ZeRO optimization, or model parallelism must supplement data parallelism.
• Practical concerns: Data loading, straggler GPUs, batch size tuning, and fault tolerance all require attention in production distributed training setups.

The next chapters cover complementary strategies that work alongside data parallelism. Tensor parallelism (Chapter 4) splits individual layers across GPUs within a node, reducing per-GPU memory proportionally. Pipeline parallelism (Chapter 5) stages model layers across sets of GPUs, enabling even larger models. ZeRO optimization (Chapter 6) achieves memory reduction equivalent to model parallelism while retaining the simpler programming model of data parallelism. Together, these form the parallelism stack used by every large-scale language model training run today.