Expert Networks › Summary

Expert networks form the computational substrate of Mixture of Experts models. The key insights from this chapter include:

• Experts are independent FFNs that mirror standard transformer feed-forward layers. The position-independence of FFN operations makes them natural candidates for sparse expert-based routing, while attention layers are poor candidates because they require cross-position communication that cannot be easily distributed across independent subnetworks.

• The SwiGLU architecture used in modern expert implementations adds a learned, input-dependent gating mechanism that allows experts to selectively filter their intermediate representations. This makes experts more expressive than simple two-layer FFNs and better suited to developing specialized transformations for narrow input distributions.

• Expert capacity manages the tension between flexible routing and efficient batched computation. The capacity factor determines how much routing imbalance the system tolerates before dropping tokens. The right capacity factor balances memory efficiency against routing quality, and monitoring drop rates during training is needed for diagnosing routing health.

• Expert count involves tradeoffs between specialization potential, memory requirements, and utilization efficiency. Successful models range from 8 experts (Mixtral) to thousands (Switch Transformer). More experts means more capacity but also more memory, more communication overhead in distributed settings, and greater risk of expert underutilization.

• Expert placement typically replaces FFN layers while keeping attention layers dense. Alternating dense and MoE layers is a common pattern that balances routing overhead with expert specialization, and this approach has been validated empirically across multiple major MoE architectures.

• Practical limitations of expert networks include memory-compute asymmetry, expert collapse (underutilization feedback loops), non-differentiable capacity constraints, and inference-time load imbalance. These challenges motivate the auxiliary losses and deployment engineering techniques covered in subsequent chapters.

With the expert architecture established, the natural question becomes: how does the model decide which expert should process each token? The next chapter on gating networks addresses exactly this question, introducing the routing mechanisms that make sparse expert selection possible and the loss functions that keep routing balanced during training.