Expert Networks › Expert Networks

In the previous chapter, we explored how sparse models achieve computational efficiency by activating only a subset of parameters for each input. The Mixture of Experts (MoE) architecture realizes this principle through a collection of specialized subnetworks, each designed to handle different aspects of the input distribution. These subnetworks, called experts, form the computational backbone of MoE models. Understanding their structure, their relationship to the classical feed-forward network, and the engineering constraints that govern how they handle batches of tokens is the focus of this chapter.

Think of the overall MoE layer as a library staffed by specialists. When a patron arrives with a question, a receptionist (the router) glances at their request and directs them to the most relevant expert in the building. Each expert in that library has spent years developing deep knowledge in a specific domain: one handles legal questions, another handles medical topics, another handles creative writing, and so on. The patron walks directly to the right desk, gets a highly informed answer, and leaves. No single patron talks to every specialist; each consultation draws on exactly the expertise relevant to that person's needs. The MoE layer works the same way: each token is a patron, each expert is a domain specialist, and the router ensures that every token reaches the expert most likely to transform it usefully.

This analogy captures the aspirational design of MoE, but the implementation details matter enormously. What exactly is the internal structure of each expert? Why is the feed-forward network the natural choice rather than, say, a small attention module? How does the model manage the computational challenge that different tokens take different paths through the network during batch processing? And how many experts should a model have, and where in the transformer stack should they be placed? These questions have real consequences for model quality, memory requirements, and training stability. Working through them carefully gives you the conceptual toolkit to understand every major MoE model in the literature, from Switch Transformer to Mixtral to DeepSeek.

A useful frame throughout this chapter is the tension between capacity and computation. Every parameter we add to a neural network increases its capacity to model complex patterns in data. But every parameter we invoke during a forward pass increases the computational cost of that pass. In dense transformers, these two quantities are locked together: more capacity means more compute, always. MoE breaks this lock. By maintaining a large pool of expert weights and activating only a small fraction per token, the architecture decouples the model's total parameter count from its per-token floating-point operations. The experts are the mechanism by which this decoupling is achieved, so understanding their design is central to understanding why MoE works at all.

This chapter examines what experts are, how they are structured, and the architectural decisions that determine their effectiveness. We will see why the feed-forward network became the dominant choice for expert implementation, how to manage the flow of tokens through experts during batch processing, and where to place expert layers within the transformer architecture. We will also examine the parameter-counting arithmetic that makes MoE scaling attractive, work through a numerical example of capacity-constrained routing, and discuss the practical limitations that arise when deploying expert-based models. By the end, you will have a thorough understanding of the building blocks that make sparse expert models function.