Expert Parallelism › The Distribution Challenge

To understand why expert parallelism is necessary, consider a concrete scenario. You are training an MoE model with 64 experts, where each expert is a two-layer feed-forward network with hidden dimension 4096. This is a common configuration: each expert acts as an independent specialist, capable of processing any token but trained to excel on a particular subset of the input space. With a 4x expansion factor in the feed-forward hidden dimension (a standard choice in transformer architectures), each expert consists of two linear transformations, one projecting up to 16384 dimensions and another projecting back down to 4096.

The parameter count scales rapidly with these choices. Each of the two linear layers in an expert contains the product of its input and output dimensions in weights. Summing across both layers, each expert holds approximately 134 million parameters. That number may sound manageable in isolation, but when you multiply by 64 experts, the expert layers alone require storing 8.6 billion parameters. And this is just the expert component: the model also has attention layers and layer normalization, plus embeddings and positional encodings, all of which add to the total. A realistic large-scale MoE model might have between 50 billion and several trillion total parameters, making the hardware requirements extraordinary.

Only 2 of the 64 experts activate for any given token in a top-2 routing scheme, but all 64 experts must exist in memory during training. You cannot materialize only the experts needed for a batch because gradient computation requires the entire parameter set to be accessible. The full 8.6 billion expert parameters must reside somewhere in the cluster's memory at all times during training. No single GPU can efficiently hold all of this. Even the NVIDIA H100 with 80 GB of HBM3 cannot hold it alone once optimizer states and activations are included alongside non-expert components. Distributing computation lets each device focus on a subset of experts and run expert computation in parallel, reducing per-step training time.

The mathematics of this situation follows directly from the architecture choices. Each expert's parameter count comes from two linear transformations: one that projects from the model dimension to the feed-forward hidden dimension, and another that projects back. When you multiply model dimension by the expanded hidden dimension for each layer and then by the number of experts, the total parameter count grows rapidly. For sufficiently large MoE models, expert parallelism is a hardware necessity rather than an optional optimization.

With 8 GPUs, expert parallelism places 8 experts on each device. This is the core principle of the strategy: partition the expert pool across the available devices rather than replicating it. Instead of asking each device to maintain a copy of every expert, which would be both memory-prohibitive and wasteful of network bandwidth for gradient synchronization, we ask each device to host a specific subset of experts. Tokens that need those experts must travel to that device, but the memory burden is distributed evenly across the cluster. The router (gating network) is replicated on all devices because it is small and must be accessible everywhere to make routing decisions, but the experts themselves each live on exactly one device.

This design choice means that communication in expert parallelism differs fundamentally from other forms of parallelism. In data parallelism, communication happens during the backward pass when gradients are aggregated across replicas; the forward pass requires no inter-device communication. In expert parallelism, communication happens during the forward pass whenever a token needs to visit an expert on a different device. This makes expert parallelism sensitive to network bandwidth and prevents communication from being hidden behind computation as easily as gradient synchronization.