Expert Networks › Expert Count Selection

Choosing the number of experts involves balancing specialization potential, memory requirements, and computational efficiency. Different models have explored configurations ranging from 8 to thousands of experts, and there is no universally optimal number. The right choice depends on the training data distribution, available hardware, and target use case. Understanding the tradeoffs qualitatively helps you reason about architectural decisions in the models you read about and build.

Think of expert count selection as deciding how many specialized workshops to maintain in a factory. A factory with 4 specialized workshops can route different types of work to appropriate stations, but the specialization is relatively coarse. A factory with 64 workshops can be much more finely specialized, with each workshop handling a narrow type of task extremely well. But maintaining 64 workshops requires more floor space, more staff, and more coordination overhead. At some point, adding more workshops provides diminishing returns if there is not enough work of each specialized type to keep every workshop productive. The optimal number of workshops depends on how diverse the incoming work is and how much floor space (memory) you have available.

The choice of expert count is also intimately connected to the routing topology. A model with 8 experts and top-2 routing activates 25% of its experts per token. A model with 64 experts and top-2 routing activates only about 3% of its experts per token. The second model has much greater sparsity, which means each expert sees a much smaller fraction of the training distribution. This can lead to richer specialization but also to undertraining risks if some experts see too few tokens to learn effective transformations. The relationship between expert count, routing sparsity, and training dynamics is an active area of research, and the empirical findings from major MoE papers provide useful reference points.

Increasing the number of experts has several competing effects that you need to weigh against each other. More experts generally improves the model's capacity to specialize, which can improve quality on diverse tasks. But it also increases the engineering and systems complexity of training and deployment.

Benefits of more experts:

• Greater total parameter count without proportional compute increase, giving the model more capacity to represent complex functions
• Finer-grained specialization potential, letting experts to develop highly targeted transformations for narrow sub-distributions
• Better scaling on distributed systems with expert parallelism, since more experts can be spread across more devices

Costs of more experts:

• More memory required to store all expert weights, even those not activated for a given token
• Communication overhead when experts span multiple devices, requiring tokens to be transferred to the device holding the relevant expert
• Risk of some experts being chronically underutilized, leading to undertrained weights that hurt quality when they are eventually selected

The communication overhead point deserves special attention because it is often the binding constraint in practice. In a model where each expert lives on a separate device (the "expert parallelism" strategy), every routing decision that sends a token to an expert on a different device requires network communication. With 8 experts on 8 devices, routing is relatively simple: most tokens stay on their local device or transfer to one of a small number of remote devices. With 1024 experts on 64 devices, routing becomes a complex all-to-all communication problem. The time spent on this communication can easily dominate the time spent on actual computation, eliminating the efficiency gains that motivated MoE in the first place.

Major MoE models have converged on different expert counts based on their design goals and available hardware:

Switch Transformer demonstrated that even single-expert routing (top-1, meaning only one active expert per token) works well when the expert count is very large, letting massive parameter counts while keeping per-token compute low. The original Switch Transformer scaled to over a trillion parameters by using thousands of experts per layer. Mixtral showed that a much simpler configuration (8 experts, top-2 routing) achieves strong quality while remaining practical to deploy on commodity hardware. DeepSeek-MoE explored a middle ground, using many fine-grained experts (64) with a higher active count (6) to capture more routing granularity.

More experts with fewer active per token increases the capacity-to-compute ratio. A model with 64 experts but only 2 active has 32 times more parameters than its per-token compute would suggest. This ratio affects:

• Memory requirements: all expert weights must be stored in memory (or on the same device in a distributed setting), even those that will not be used for any given batch
• Potential for overfitting: more capacity than the training data can support leads to experts that memorize training patterns rather than learning generalizable transformations
• Inference efficiency: expert weights may need to be loaded from slower memory (system RAM or disk) if the GPU memory cannot hold all of them simultaneously

The key insight here is that the right capacity-to-compute ratio depends on the training data volume as much as the hardware configuration. A model trained on 1 trillion tokens can productively use much higher capacity than a model trained on 100 billion tokens, because there is sufficient diversity in the data to train every expert to do something useful. Scaling laws for MoE are an active research area, and the relationship between expert count, training tokens, and final quality is not yet as well-characterized as for dense models.

Some architectures include "shared" experts that process every token alongside the sparse routing to specialized experts. DeepSeek-MoE uses this approach, with 2 shared experts plus routing among 64 specialized experts. Shared experts capture common transformations that benefit every token, such as basic syntactic processing or common lexical patterns, while specialized experts handle token-specific or domain-specific processing.

The intuition for shared experts comes from the observation that not all processing is specialized. Every sentence, regardless of its domain, involves basic grammatical relationships, function-word handling, and common semantic operations that benefit from consistent treatment. If specialized experts learn these common operations through repeated routing of common tokens, they may not have sufficient capacity left for specialization. Dedicated shared experts can handle these universal computations, leaving specialized experts to focus on differences between domains and token types.