Communication Optimization › Topology-Aware Communication

The physical network connecting GPUs in a cluster is not a flat, uniform fabric. Understanding its structure, and mapping collective communication algorithms onto it accordingly, can dramatically reduce latency and increase effective bandwidth.

A typical GPU cluster has several levels of connectivity, each with different bandwidth and latency characteristics.

Within a single node, GPUs are connected via NVLink (or PCIe on older hardware). An A100-class NVLink 3.0 fabric provides up to about 600 GB/s of aggregate bidirectional bandwidth per GPU, with near-zero latency because the links are direct electrical connections. At that aggregate rate, moving a 1 GB gradient tensor takes under 2 milliseconds.

Between nodes, GPUs communicate over InfiniBand HDR or Ethernet. InfiniBand offers approximately 200 Gb/s (25 GB/s) per link with latency around 1-3 microseconds. But this bandwidth is shared among all GPU-to-GPU transfers traversing that link, and the effective bandwidth per GPU decreases as the cluster size grows.

Within a large cluster, nodes are connected through one or more layers of network switches. A 3-level fat-tree topology with 100 nodes might have edge switches connecting groups of 10 nodes, aggregation switches connecting groups of edge switches, and a core switch layer. Bandwidth at the edge layer is full-bisection, but bandwidth at higher layers may be oversubscribed by 2x or 4x, meaning communications that cross the top-of-rack switches run at half or quarter the link speed.

The implication for collective algorithms is clear: all-reduce operations that keep communication within a node use NVLink and are essentially free. Operations that span nodes use InfiniBand and are moderately expensive. Operations that cross the aggregation layer become expensive. A topology-unaware all-reduce that routes traffic randomly through the cluster may cause some inter-node links to carry far more traffic than others, resulting in congestion and underutilization.

Another consequence of hierarchical topology is that all-reduce latency scales differently at each level. Intra-node latency is dominated by CUDA kernel launch overhead (microseconds). Inter-node latency is dominated by InfiniBand message transmission (tens to hundreds of microseconds for large messages). Aggregation-layer latency can involve multiple hops, each adding latency. For very large collectives spanning thousands of GPUs, the latency to complete a single all-reduce even at peak bandwidth can become significant, motivating research into asynchronous and pipelined alternatives.

The practical effect of ignoring topology can be severe. Consider a 64-GPU job on 8 nodes, using a flat ring all-reduce that treats all 64 GPUs symmetrically. The ring algorithm will assign adjacent GPUs in the ring to potentially be on different nodes, meaning a significant fraction of the ring's traffic crosses slow inter-node links even when the data could have traveled entirely within a single node. A topology-aware implementation avoids this by ensuring that ring neighbors that communicate frequently are on the same node.

The standard ring all-reduce algorithm arranges all $N$ workers in a ring and passes gradient chunks around the ring in two phases: a reduce-scatter phase that computes partial sums, and an all-gather phase that distributes the final sums. This algorithm achieves optimal bandwidth ($2(N-1)/N$ of theoretical peak) and is topology-agnostic.

Topology-aware implementations improve on this by exploiting the bandwidth hierarchy. A hierarchical all-reduce proceeds as follows:

1. Within each node, perform a local reduce-scatter using NVLink at full intra-node bandwidth
2. Each node now holds a partial sum for $1/N_{\text{nodes}}$ of the parameters
3. Across nodes, perform a node-level all-reduce using InfiniBand, exchanging only partial sums
4. Each node now holds a complete sum for its slice of parameters
5. Within each node, perform a local all-gather using NVLink to distribute the complete sums

This two-level approach concentrates the bulk of data movement onto the faster intra-node fabric. In step 3, each node sends only $P / N_{\text{nodes}}$ parameters over InfiniBand instead of all $P$ parameters, reducing inter-node traffic by a factor of $N_{\text{nodes}}$.

For a cluster with 32 nodes of 8 GPUs each, hierarchical all-reduce reduces inter-node communication by 32x compared to a flat ring algorithm, while using the full NVLink bandwidth for the remainder. The speedup in practice is not quite 32x because of the added intra-node phases, but the dominant cost (inter-node InfiniBand transfer) is indeed reduced by this factor.

NCCL detects NVLink and NVSwitch topology automatically using CUDA topology queries and the NCCL topology detection protocol. On a properly configured DGX node, NCCL will use NVLink for intra-node phases without requiring manual configuration. The topology-aware optimization is thus "free" in the sense that it happens automatically, but it requires NVLink-connected hardware rather than PCIe-connected devices within a shared chassis.

The algorithm correctness is easy to verify. After the intra-node reduce-scatter, node $i$ holds a vector $v_i^{(s)}$, the partial sum of all gradients from GPUs on that node, for the $s$-th slice of the parameter space. After the inter-node all-reduce in step 3, every node has $\sum_i v_i^{(s)}$, the complete sum across all nodes, for its slice $s$. After the intra-node all-gather in step 5, every GPU on every node has the complete sum for every slice. The final state is identical to a flat ring all-reduce, achieved with far less inter-node traffic.

An even more aggressive topology optimization offloads the actual reduce computation to network switches. NVIDIA's SHARP (Scalable Hierarchical Aggregation and Reduction Protocol) places reduction logic directly into InfiniBand switches. Instead of each GPU sending data to every other GPU with the switch acting as a passive fabric, SHARP-enabled switches perform the partial sum computation in-flight.

With SHARP, an all-reduce over $N$ nodes transfers only $P / N_{\text{switches}}$ bytes per switch link, with switches in the aggregation layer combining data from multiple lower-level switches before passing it upward. The leaf switches collect gradient shards from their connected GPUs, sum them, and pass only the sum to the parent switch. The parent switch sums contributions from multiple leaf switches and passes the result upward. Data volume decreases as you move up the switch hierarchy, and the final all-gathered result flows back down.

SHARP requires specific hardware support and careful configuration, but the performance gains can be substantial in large clusters. MLPerf training benchmarks have demonstrated 30-40% improvements in all-reduce bandwidth with SHARP enabled. The key insight is that by computing partial sums in the switch fabric itself, you eliminate redundant traffic: without SHARP, every intermediate partial sum flows from its origin GPU across the entire network before being aggregated; with SHARP, partial sums are combined at the first switch that sees all contributions, and only the final result propagates further.

The main limitation of SHARP is that it requires the entire switch path to support the protocol. A single non-SHARP switch in the path disables SHARP for that communication, falling back to standard all-reduce. This means SHARP benefits are most reliable in purpose-built clusters with uniform hardware. Cloud environments with mixed switch generations may not reliably enable SHARP.