Expert Parallelism › Communication Overhead Analysis

Expert parallelism introduces communication costs that do not exist in dense models. Understanding these costs is needed for system design and performance optimization. While dense model training also involves communication (for gradient synchronization, for example), the all-to-all operations in MoE occur in the forward pass itself, directly adding latency to every training step. Unlike gradient synchronization, which can often be overlapped with the backward pass computation through techniques like gradient bucketing, the dispatch and combine all-to-all operations in the forward pass sit on the necessary path: the expert computation cannot begin until the dispatch is complete, and the next layer cannot begin until the combine is complete.

This is a subtle but important distinction. In data-parallel training with gradient synchronization, the communication is largely "free" from a latency perspective because it happens in the background while computation continues. In expert parallelism, the forward-pass communication creates unavoidable serial dependencies. The training step time is therefore: routing computation, plus dispatch all-to-all, plus expert computation, plus combine all-to-all, plus remaining layer computation. The all-to-all operations add directly to the necessary path length, not just to bandwidth utilization.

For each MoE layer, we perform two all-to-all operations. The first operation dispatches tokens from their originating devices to the devices holding their target experts. The second operation returns the processed results back to the originating devices. Each operation moves the same amount of data, as every token that goes out must come back. The total communication volume is therefore symmetric: dispatch and combine are mirror images of each other.

The total communication volume $V_{\text{layer}}$ (in elements) per MoE layer can be expressed in terms of the key configuration parameters. We compute this by multiplying the number of token-expert pairs by the size of each token representation, then doubling for the two communication phases:

$$V_{\text{layer}} = 2 \times B \times S \times K \times D$$

where:

• $B$: the batch size (number of sequences). Larger batches increase throughput but also increase the total number of tokens that must be routed.
• $S$: the sequence length (number of tokens per sequence). Longer sequences mean more tokens per batch, directly scaling communication.
• $K$: the top-$k$ parameter (number of experts per token). When $k=2$, each token creates two communication events rather than one.
• $D$: the hidden dimension (size of each token vector). Larger models with wider hidden dimensions must send more data per token.
• $2$: accounts for the two all-to-all operations (dispatch and combine). This factor is fixed by the algorithm structure.

Why does this formula make sense? Notice that $B \times S$ gives the total number of tokens in the batch. Multiplying by $K$ gives the total number of token-expert pairs, since each token creates $K$ routing events. Multiplying by $D$ converts token count to element count, since each token is a vector of $D$ elements. The factor of 2 reflects that each element travels twice: once to the expert device and once back. The multiplicative relationship between these factors means that communication volume grows rapidly with any of the individual parameters. Doubling any single factor doubles the communication cost.

This substantial volume of data movement, roughly 32 GB per forward pass for a single batch, highlights why network bandwidth is often the bottleneck in MoE training. To put this in perspective, even with high-bandwidth interconnects capable of 200 GB/s, transferring 32 GB takes around 160 milliseconds if the transfers were purely sequential. In practice, the all-to-all implementation uses bidirectional transfers and pipeline overlap to reduce effective latency, but the basic constraint remains: you cannot transfer 32 GB faster than the interconnect bandwidth allows. The communication cost scales linearly with the number of MoE layers, making efficient routing needed. This is one reason why some architectures use MoE layers only in alternating positions rather than in every layer, reducing the number of all-to-all operations by half at the cost of slightly reduced expert capacity.

The key metric for understanding expert-parallelism efficiency is the ratio of communication time to computation time. If communication dominates, adding more devices provides diminishing returns because each device spends more time waiting for data than computing. Conversely, if computation dominates, the system can efficiently use additional devices for expert computation without communication becoming a bottleneck.

This ratio is determined by what is sometimes called the arithmetic intensity of the all-to-all: the ratio of compute operations performed to bytes transferred. For expert parallelism, we transfer token representations to expert devices and get back transformed representations. The computation performed on those representations is the expert forward pass, which involves two matrix multiplications per token. The ratio of FLOP to bytes transferred determines whether the system is compute-bound or bandwidth-bound.

As we scale to more devices, the fraction of tokens that must cross device boundaries increases. With $N$ devices, approximately $(N-1)/N$ of all token-expert pairs must be transferred across the network (assuming uniform routing). As $N$ grows, this fraction approaches 1, meaning almost all communication is cross-device. This is why communication overhead grows with device count: more devices means more cross-device traffic for the same total token count.