Gating Networks › Limitations and Impact

The gating network design we've covered is the standard approach used in models like Switch Transformer and Mixtral. While effective, this design has important limitations that shape MoE research and practice.

Routers make discrete decisions while training requires continuous gradients. The standard approach handles this by flowing gradients through routing weights, not expert selection. This works but creates blind spots: the model never learns whether unselected experts might have been better choices. Several techniques address this limitation. Load balancing losses encourage the router to explore all experts by penalizing uneven usage. Random routing occasionally computes non-selected experts for gradient purposes. And capacity factors limit how many tokens any single expert can handle, forcing distribution across experts. We'll explore these techniques in the upcoming chapters on load balancing and auxiliary losses.

Another limitation is the simplicity of the routing decision. A single linear layer might not capture complex routing logic. Some research explores hierarchical routers (first choose an expert cluster, then an expert within it) or multi-layer routers. However, the simple linear approach remains dominant because more complex routers add overhead and don't consistently improve performance.

Despite these limitations, gating networks let sparse expert models match or exceed dense-model quality at lower computational cost. The router learns specialization by routing different content types to appropriate experts, showing that simple gating mechanisms can capture structure in language. This specialization emerges from end-to-end training without explicit supervision about which expert should handle which content. The router discovers divisions of labor by learning which expert produces better outputs for each input type.