Expert Parallelism › Key Parameters

The key parameters for the Expert Parallel MoE implementation shape both performance and model quality. Understanding what each parameter controls helps you make informed choices when designing a new MoE configuration or debugging an existing one.

The first parameter to consider is num_experts, the total number of experts in the model. This determines the potential for specialization and affects both model capacity and communication patterns. Common values range from 8 (as in Mixtral-8x7B) to hundreds or even thousands (as in Switch Transformer and its descendants). More experts provide greater capacity and finer-grained specialization opportunities, but also increase the probability of routing imbalances and the complexity of the all-to-all communication pattern.

The parameter num_local_experts specifies how many experts reside on the current device, typically num_experts / world_size. This value determines the memory footprint per device and must divide evenly into the total expert count for uniform placement. If your cluster has 8 GPUs and you want 64 experts, each GPU holds 8 local experts. This ratio, the number of experts per device, also affects the expected fraction of local routing: with $L$ local experts out of $E$ total, on average $L/E$ of all token-expert pairs route locally without requiring network communication.

The top_k parameter, typically 1 or 2 in production models, controls how many experts each token visits. The Switch Transformer used top-1 to minimize communication overhead; subsequent work showed that top-2 provides better model quality at a modest increase in communication cost. Values above 2 are rare in practice because the marginal quality improvement diminishes while the communication cost grows linearly. Some recent architectures explore more complex routing schemes, such as always routing to one "shared" expert and using top-k for the remaining selections, which provides a floor of universal computation while retaining selective specialization.

The capacity_factor parameter reserves extra buffer space for load imbalances, typically set to values between 1.0 and 1.5. A value of 1.25 means each expert is pre-allocated 25% more capacity than the expected average load. This headroom absorbs natural variation in routing patterns, preventing token dropping when some experts receive more traffic than expected in a given batch. The tradeoff is memory: higher capacity factors consume more memory per expert, reducing the maximum batch size you can use. When tokens exceed expert capacity, they are typically dropped (their contribution to the output is zeroed), creating a training signal artifact that can harm model quality if the dropping rate is high.