Gating Networks › Summary

Gating networks serve as the decision-making component in Mixture of Experts architectures, determining which experts process each token. This chapter covers:

• Router architecture: Standard routers use a single linear layer projecting token representations to expert scores. This simple design balances computational efficiency with sufficient expressivity.

• Routing score computation: Raw logits pass through softmax to produce a probability distribution over experts. Top-K selection then identifies the experts used for computation, with renormalized weights for combining their outputs.

• Router training: Gradients flow through routing weights (which are continuous) even though expert selection is discrete. This creates blind spots for unselected experts, motivating auxiliary training techniques.

• Learned behavior: Trained routers exhibit real specialization, including semantic domain preferences, syntactic patterns, and layer-dependent routing strategies. This specialization emerges naturally from end-to-end training.

The next chapter covers Top-K routing in detail, examining how the number of selected experts affects model capacity, computation costs, and training dynamics.