Expert Networks › Limitations and Practical Considerations

Expert networks introduce several practical challenges. The most immediate is memory overhead: storing $N$ experts requires $N$ times the memory of a single FFN layer, even though most experts remain idle for any given token. A Mixtral 8x7B model stores 8 separate FFN copies per MoE layer, consuming significant GPU memory despite only activating 2 at inference time. This memory-compute asymmetry means MoE models often require more hardware to deploy than their per-token FLOP count suggests. A model computationally equivalent to a 7B dense model at inference time may require the memory bandwidth and storage of a 47B parameter model because all expert weights must be resident in memory.

This memory-compute split creates practical deployment challenges that do not arise with dense models. For dense models, memory and compute scale together: a model that needs more memory also needs more compute, and hardware allocation is relatively straightforward. For MoE models, you must separately optimize for peak memory (which is determined by all expert weights) and throughput (which is determined by active expert weights). Quantizing expert weights to lower precision (int4 or int8) is one common approach to managing this tradeoff, since quantization reduces memory footprint with relatively modest quality degradation for the inactive experts.

Expert underutilization presents another concern. Without careful load balancing, some experts may receive far fewer tokens during training, leading to undertrained weights that hurt model quality when they are eventually selected. This creates a feedback loop: underutilized experts produce worse outputs, causing the router to select them even less frequently. The auxiliary losses we will discuss in upcoming chapters directly address this problem by penalizing routing distributions that are too concentrated. The severity of this collapse behavior depends on the routing mechanism and the training hyperparameters, and it remains one of the more delicate aspects of MoE training.

The capacity mechanism itself introduces discontinuities in the training signal. Tokens dropped due to capacity overflow still contribute to the loss but receive no expert processing, creating a mismatch between forward computation and gradient flow. The router does not receive a gradient signal telling it that certain tokens were dropped: from the perspective of the loss function, those tokens simply passed through the residual connection, which is a valid (if suboptimal) outcome. Various techniques attempt to make this process more differentiable, but some information loss is inherent to hard capacity constraints. This is one reason why some recent MoE variants have explored soft capacity mechanisms and token merging approaches that avoid hard dropping entirely.

Load balancing at inference time is also a non-trivial concern. During training, the model sees a diverse mixture of inputs, and the routing distribution averages out over many batches. At inference time, however, you may process a stream of requests that are all from the same domain: a coding assistant receiving only code inputs, for example, may route almost all tokens to a small subset of experts. This request-time imbalance can cause expert parallelism bottlenecks that did not appear during training, degrading inference throughput. Some deployment systems address this by routing similar requests together to avoid within-batch imbalance, but this adds infrastructure complexity.

Finally, expert placement decisions remain somewhat empirical. While alternating dense and MoE layers has become standard, the optimal interleaving pattern likely varies by task and scale. Some recent work suggests that certain layer positions benefit more from expert diversity than others, but clear design principles are still emerging from the research community. The field does not yet have the equivalent of scaling laws for MoE placement: we know that some configurations work well empirically, but a principled theory of why certain placements outperform others is still an open question.