Communication Optimization › Key Parameters
Michael BrenndoerferJanuary 23, 20261 min read
Language AI HandbookAll chapters Part 31 of 67Training Infrastructure
Communication Optimization › Key ParametersOne section · Sections in this chapter7 / 10 ↓
Sections in this chapter10
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The key parameters for communication optimization are:
- Compression ratio: The fraction of gradients transmitted per step (top-K) or the bit-width ratio (quantization). Lower ratios reduce communication but increase the risk of convergence degradation without error feedback.
- Bucket size (DDP): Larger buckets reduce all-reduce overhead per operation but decrease the granularity of overlap. PyTorch default is 25 MB; larger models often benefit from values of 100-500 MB.
- NCCL buffer size (
NCCL_BUFFSIZE): Larger buffers increase throughput for large all-reduces at the cost of GPU memory. Typical useful range: 4 MB to 64 MB. - NCCL channels (
NCCL_NCHANNELS): More DMA channels increase available bandwidth on high-speed interconnects. Range: 2-32; match to physical port count. - Rank (PowerSGD): The low-rank approximation rank . Higher rank preserves more gradient information but reduces compression. Typical values: .
- Error feedback coefficient: Whether accumulated residuals are scaled before addition to the next step's gradient. Values below 1.0 can stabilize training with aggressive compression but reduce the effective "memory" of the accumulator.