Expert Parallelism › Expert Parallelism

Mixture of Experts models achieve their remarkable efficiency by activating only a subset of parameters for each token. As we explored in previous chapters, a model with 64 experts might only use 2 experts per token, giving you the capacity of a massive model with the compute cost of a much smaller one. A single forward pass through the model touches only a fraction of the total parameter count, yet the model has access to all of the capacity when it needs it. This conditional computation property is what makes MoE architectures so appealing: you pay for what you use, not for everything you have.

But this creates a basic systems challenge that becomes acute at scale. As models grow to hundreds of billions or even trillions of parameters spread across dozens of expert networks, where do all those experts physically reside? No single GPU can hold them all. And how do tokens get to the right experts when those experts might live on a completely different device, possibly in a different server rack across the data center? The clever routing mechanism that makes MoE efficient at the algorithmic level creates a deep systems problem at the hardware level.

Expert parallelism is the distributed-computing strategy designed specifically to solve this problem. It is the engineering foundation that makes large-scale MoE training possible. Unlike data parallelism, which replicates the entire model across devices and processes different data on each copy, or tensor parallelism, which shards individual weight matrices across devices, expert parallelism distributes the expert networks themselves. Each device in an expert-parallel setup becomes responsible for a specific subset of experts. Tokens travel to wherever their selected experts live, computation happens locally on the hosting device, and results travel back. This chapter examines how this works in practice: from assigning experts to devices, through the all-to-all communication patterns that move tokens across the cluster, to the implementation details that determine hardware utilization.

Understanding expert parallelism requires thinking simultaneously at two levels: the algorithmic level, where we reason about routing decisions and expert computations, and the systems level, where we reason about network bandwidth and latency as well as hardware utilization. The two levels are deeply interconnected. A routing decision that looks elegant at the algorithmic level can be a performance disaster at the systems level if it creates lopsided communication traffic. Conversely, systems optimizations like capacity factors and token dropping have direct implications for model quality. Getting expert parallelism right means understanding both levels and the tensions between them.

Think of expert parallelism like a library system distributed across multiple branch locations. Each branch specializes in different subjects, holding the relevant books. When a patron submits a research request (a token), a routing coordinator (the gating network) determines which branches have the most relevant materials (which experts to consult). The patron's request travels to those branches, the relevant books are retrieved, and the results come back to the patron. The coordinator does not need to know the contents of every book; it just needs to know which branches specialize in what. The efficiency of this system depends critically on how well the branches are stocked relative to demand, how quickly requests can travel between locations, and whether popular branches become overwhelmed with requests while others sit idle.