Gating Networks › Gating Networks

In the previous chapter, we explored how expert networks give specialized computational pathways within a Mixture of Experts (MoE) architecture. But having multiple experts raises an immediate question: how does the model decide which expert to use for each input? This decision falls to the gating network, also called the router, which is the traffic controller directing tokens to appropriate experts.

The router might seem like a minor component (often just a single linear layer), but its design affects model quality and training stability. A poorly designed router can cause expert collapse (where only one expert gets used), severe load imbalance, or failure to learn specialization. Getting the router right is one of the central challenges in MoE systems.

This chapter examines how routers work: their architecture, how they compute routing scores, how they learn through backpropagation, and what behavioral patterns emerge during training. Understanding the router gives the basis for the routing strategies and balancing techniques we'll explore in upcoming chapters.