Expert Parallelism › Summary

Expert parallelism is the distributed computing strategy that makes large-scale MoE models practical. Rather than replicating all experts on every device (which is memory-prohibitive) or sharding experts arbitrarily (which creates complex parameter ownership), expert parallelism assigns complete experts to devices and uses all-to-all communication to route tokens to wherever their selected experts reside. The strategy works because expert computation is independent across experts: once a device receives all tokens destined for its experts, it can process them locally without further coordination, and the results can be returned asynchronously via the second all-to-all.

The core concepts work together in a pipeline: expert placement assigns experts to devices at initialization, setting up the mapping that routing decisions will use. The gating network makes routing decisions in the forward pass, creating expert indices for each token. All-to-all dispatch sends tokens to their target devices; expert computation processes them locally; all-to-all combine returns results to originating devices. The entire pipeline repeats for every MoE layer in the model and every batch during training.

The key considerations for designing an expert-parallel system include:

• Expert placement assigns experts to devices, typically uniformly, though capacity-aware placement can improve load balance when expert utilization patterns are known or can be estimated from prior runs.
• All-to-all communication is the primitive that enables dynamic routing, where each device sends different data to each other device based on the gating network's decisions. The two all-to-all operations per MoE layer sit on the necessary path and cannot be hidden behind computation.
• Communication overhead grows with device count and can become a bottleneck at scale, making high-bandwidth interconnects needed for maintaining training efficiency. Practical systems often limit expert parallelism to within-node communication to exploit NVLink bandwidth.
• Implementation requires careful buffer management for variable-length messages and coordination with routing decisions to preserve token ordering through the dispatch-compute-combine pipeline.

Expert parallelism combines naturally with other parallelism strategies. Production systems typically use expert parallelism for the MoE layers alongside data parallelism for throughput and potentially tensor parallelism for very large expert dimensions. The choice of which strategies to combine, and in what proportions, depends on the specific model architecture, available hardware, and performance requirements. Getting the combination right is one of the central engineering challenges in large-scale MoE training, and it rewards careful analysis of the communication and computation time breakdowns for the specific configuration at hand.