Load Balancing › The Expert Utilization Problem

When a gating network routes tokens to experts, nothing inherently prevents it from developing a preference for certain experts over others. In fact, the opposite is true: the training dynamics of MoE systems naturally push toward imbalanced routing. This tendency emerges not from any flaw in the architecture design but from the basic way that learning systems respond to feedback signals.

Think of the gating network's job as similar to a hiring manager who must allocate work across a team of consultants. At the start of a project, the manager has no basis for preferring one consultant over another, so work is distributed somewhat evenly. But as the project progresses, certain consultants demonstrate strong performance on early tasks. Rationally, the manager routes more important work to these consultants. The more they work, the more they develop their skills and demonstrate capability, reinforcing the manager's preference. Meanwhile, the consultants who receive less work have fewer opportunities to demonstrate their abilities, and over time their skills may even atrophy. By the end of the project, a few consultants are overworked while others sit idle, even though the original team was assembled to handle a diverse workload. The MoE gating network is this manager, and the experts are the consultants.

To understand why imbalance occurs at a mechanistic level, consider what happens during the early stages of training. If expert $E_3$ happens to perform slightly better than its peers on a few early batches, perhaps due to nothing more than favorable random initialization, the gating network learns to send more tokens to $E_3$. This makes sense from an optimization perspective: the gating network is simply learning to route tokens to whichever expert currently produces the best outputs. However, this reasonable local decision creates a problematic global pattern.

With more tokens flowing to $E_3$, this expert receives more gradient updates and accumulates more training signal. These additional updates allow $E_3$ to improve further, which in turn reinforces the gating network's preference for routing to this expert. Meanwhile, experts that receive fewer tokens get less training signal and fall further behind. They have fewer opportunities to learn from data, fewer gradient updates to refine their weights, and consequently less capability to offer when tokens are occasionally routed their way. This disparity creates a positive feedback loop that amplifies initial imbalances over time.

The key insight is that the MoE system contains a hidden instability. The gating network's training objective, to route tokens well, creates incentives that undermine the system's own capacity utilization. Every local improvement in routing quality slightly worsens the global distribution of learning. This is why load balancing cannot be an afterthought: it must be built into the training procedure from the start.

The problem manifests at two distinct but interconnected levels. At the token level, some experts process disproportionately many tokens within each batch, meaning that the computational load is not evenly distributed across the available expert networks. At the training level, some experts receive far more gradient updates over the course of training, meaning that certain experts learn extensively while others remain undertrained. Both types of imbalance waste model capacity, and together they create a situation where the theoretical advantages of MoE architectures cannot be realized in practice. The model pays the memory and complexity costs of having many experts while only benefiting from a few.

An 8-expert MoE layer with severe imbalance might route 60% of tokens to two experts while the remaining six experts share only 40% of tokens. In the extreme, several experts might receive essentially no tokens at all, sitting idle while a small subset of their peers handle all the processing workload. From the outside, the model looks like an 8-expert system. From the inside, it is a 2-expert system wearing an expensive costume.

This imbalanced state creates two distinct but equally serious problems that undermine the value of the MoE architecture:

• Wasted parameters: Underutilized experts contribute little to model predictions despite consuming memory. A model with 8 experts where only 3 are active effectively has the capacity of a 3-expert model with extra overhead. The memory budget allocated to store the weights of unused experts could have been spent on additional layers, larger hidden dimensions, or other architectural improvements that would benefit the model's performance.

• Training bottlenecks: In distributed settings where each expert resides on a different accelerator, load imbalance creates stragglers. The accelerators hosting popular experts finish processing their tokens while others sit idle, then must wait for synchronization. This waiting time represents pure waste: expensive hardware consuming power without creating useful computation.

The computational inefficiency is particularly acute in expert parallelism, where each expert is placed on a separate device. Consider a scenario where expert $E_1$ receives 10 times more tokens than expert $E_7$. In this case, the device hosting $E_1$ performs 10 times more computation while the device hosting $E_7$ mostly waits. Since training requires synchronization across all devices at regular intervals, the entire system is bottlenecked by the slowest device. Paradoxically, the slowest device in this context is not the one with the least capable hardware but rather the one hosting the most overloaded expert.

There is also an insidious interaction between load imbalance and batch size. As imbalance grows, the effective batch size for each expert diverges. The overloaded experts process more examples and receive stronger, lower-variance gradient estimates. The underloaded experts process fewer examples and receive higher-variance, noisier gradients. The resulting training dynamics are uneven: some parts of the model converge quickly and reliably while others wander due to gradient noise. This uneven convergence makes the model harder to tune and the final result harder to predict.