Switch Transformer › Switch Transformer

The previous chapters in this part established the fundamentals of Mixture of Experts: how expert networks specialize, how gating mechanisms route tokens, and how auxiliary losses maintain load balance. Yet despite these elegant solutions, MoE models remained notoriously difficult to train at scale. Training instability, communication overhead, and complex gradient interactions meant that even the most carefully designed MoE architectures struggled to reliably improve over dense baselines when pushed to the sizes that truly matter for language modeling. The Switch Transformer, introduced by Fedus, Zoph, and Shazeer in 2022, changed this by making a counterintuitive choice: route each token to just one expert instead of two or more. This radical simplification, combined with careful engineering around capacity limits and training stability, enabled scaling to over a trillion parameters while maintaining computational efficiency.

The Switch Transformer demonstrated that complexity is not always the path to better performance. By removing the interpolation between multiple experts that previous MoE designs required, Switch reduced communication overhead, simplified gradient computation, and paradoxically improved model quality. This might seem backwards at first: consulting more experts should give richer representations, right? The Switch paper showed empirically that this intuition breaks down at scale, because the benefits of simpler, cleaner optimization outweigh the representational gains of multi-expert interpolation. Understanding why this works reveals something deep about how sparse models behave during training.

Think of top-1 routing like assigning each student in a class to exactly one specialist tutor for a given question, rather than having them consult two tutors and average the advice. At first glance, two tutors sounds better. But in practice, assigning clear responsibility to one tutor means the tutor learns the student's needs and can be held accountable for the outcome. Two tutors can disagree, dilute each other's signal, or simply duplicate effort. The clarity of single-expert assignment can matter more than the breadth of multi-expert consultation.

This chapter examines the design decisions behind Switch Transformer: the Switch layer that replaces standard feed-forward networks, the top-1 routing mechanism and why it outperforms top-2, the capacity factor system that handles routing imbalances, the training stability techniques that make trillion-parameter training feasible, and the scaling results that established MoE as a path toward larger language models. We also examine fine-tuning challenges, distillation strategies, and limitations that subsequent architectures have addressed.