Mixture-of-Experts at Scale (2021) › Limitations

Despite their advantages, MoE architectures introduced several new challenges that researchers had to address. One significant limitation was the complexity of training MoE models compared to dense models. The routing mechanism added an extra component that needed to be trained, and the load balancing objectives added complexity to the training process. Training MoE models required careful tuning of hyperparameters related to routing, load balancing, and capacity constraints, making them more difficult to train effectively than dense models.

The dynamic routing in MoE models could also introduce training instability. The routing decisions depended on learned parameters, and the routing patterns could change during training as the model learned. This dynamic nature could lead to inconsistent training dynamics, where routing patterns would shift suddenly and cause training to become unstable. Researchers had to develop techniques to stabilize training, such as using auxiliary losses and carefully designed capacity constraints.

Memory requirements for MoE models were also more complex than for dense models. While MoE models could achieve better computational efficiency through sparse activation, they still needed to store all expert parameters in memory, which could be substantial for models with many experts. Additionally, the routing mechanism and load balancing computations added some memory overhead. For very large MoE models with hundreds of experts, the memory requirements could still be significant, though typically more manageable than equivalent dense models.

The routing mechanism itself introduced some computational overhead. Computing routing scores for each token and selecting top-k experts added computation that dense models didn't require. While this overhead was typically small compared to the savings from sparse activation, it did reduce some of the efficiency gains. The routing overhead became more significant for models with many experts, where computing routing scores for all experts could add noticeable computational cost.

Another limitation was that MoE models could be less predictable than dense models. Because different experts were activated for different inputs, the computational cost of inference could vary depending on which experts were selected. This variability made it more difficult to predict inference time and could complicate deployment scenarios where consistent latency was important. Dense models, by contrast, had consistent computational cost for all inputs.