Communication Optimization › Communication Optimization

When you scale a neural network training job across dozens or hundreds of GPUs, something unexpected happens: the GPUs themselves are rarely the bottleneck. The interconnects between them are. A cluster of 256 H100s can perform exaFLOPS of computation, but those operations are meaningless unless the gradient updates computed on each device can reach every other device quickly. Communication optimization is the discipline of making that data movement as fast and as invisible as possible.

This matters more than most practitioners realize. In a naive distributed training setup, gradient synchronization can account for 30-60% of total training time. Modern large language models train for months on thousands of GPUs; shaving even 10% off communication overhead translates to weeks of wall-clock time and millions of dollars in compute costs. The techniques in this chapter, ranging from lossy gradient compression to hardware-aware collective operations, represent some of the most impactful engineering work in the field.

As we covered in earlier chapters on data parallelism and model parallelism, distributed training requires coordinating parameter updates across many devices. What those chapters treated as a given, the synchronization step, is precisely what we will now take apart and optimize. The challenge is not simply "make data transfer faster." It involves mathematical approximations that deliberately throw away information, careful orchestration of computation and communication on the same hardware, awareness of the physical network topology your cluster uses, and tuning the low-level collective communication library (NCCL) that sits beneath every popular deep learning framework.

It helps to think about communication overhead in terms of the roofline model you may recall from performance analysis. A training step has a theoretical compute bound (how fast the GPU can process matrix multiplications) and a bandwidth bound (how fast data can move through the memory and network hierarchy). When communication dominates, the job is bandwidth-bound, and no amount of additional GPU capacity will help until the bandwidth bottleneck is addressed. The techniques in this chapter move the roofline: either by reducing the total data volume that must be communicated (compression), by hiding communication time behind compute (overlap), or by ensuring the hardware delivers its rated bandwidth as efficiently as possible (topology-aware communication and NCCL tuning).

Before diving into the techniques themselves, it is worth understanding why the communication problem is so hard to solve through hardware alone. InfiniBand bandwidth has improved roughly four-fold every five years, from around 10 Gb/s in 2010 to 400 Gb/s today. But model sizes have grown by orders of magnitude over the same period: GPT-2 had 1.5 billion parameters, GPT-3 had 175 billion, and the trend continues upward. The math is unforgiving. Sending 175 billion float32 parameters over a 400 Gb/s link takes nearly 14 seconds. Even with the newest hardware, you cannot train GPT-3-scale models without some form of communication optimization.

The four main strategies form a coherent stack. Gradient compression attacks the problem at the source by reducing what must be sent. Communication overlap addresses the scheduling problem by running sends and receives in parallel with backward computation. Topology-aware algorithms route traffic through the fastest available links. And NCCL tuning extracts maximum hardware efficiency from whatever link you have. Most production training systems apply all four simultaneously.