Expert Networks: MoE Architecture & FFN Implementation

In the previous chapter, we explored how sparse models achieve computational efficiency by activating only a subset of parameters for each input. The Mixture of Experts (MoE) architecture realizes this principle through a collection of specialized subnetworks, each designed to handle different aspects of the input distribution. These subnetworks, called experts, form the computational backbone of MoE models. Understanding their structure, their relationship to the classical feed-forward network, and the engineering constraints that govern how they handle batches of tokens is the focus of this chapter.

Think of the overall MoE layer as a library staffed by specialists. When a patron arrives with a question, a receptionist (the router) glances at their request and directs them to the most relevant expert in the building. Each expert in that library has spent years developing deep knowledge in a specific domain: one handles legal questions, another handles medical topics, another handles creative writing, and so on. The patron walks directly to the right desk, gets a highly informed answer, and leaves. No single patron talks to every specialist; each consultation draws on exactly the expertise relevant to that person's needs. The MoE layer works the same way: each token is a patron, each expert is a domain specialist, and the router ensures that every token reaches the expert most likely to transform it usefully.

This analogy captures the aspirational design of MoE, but the implementation details matter enormously. What exactly is the internal structure of each expert? Why is the feed-forward network the natural choice rather than, say, a small attention module? How does the model manage the computational challenge that different tokens take different paths through the network during batch processing? And how many experts should a model have, and where in the transformer stack should they be placed? These questions have real consequences for model quality, memory requirements, and training stability. Working through them carefully gives you the conceptual toolkit to understand every major MoE model in the literature, from Switch Transformer to Mixtral to DeepSeek.

A useful frame throughout this chapter is the tension between capacity and computation. Every parameter we add to a neural network increases its capacity to model complex patterns in data. But every parameter we invoke during a forward pass increases the computational cost of that pass. In dense transformers, these two quantities are locked together: more capacity means more compute, always. MoE breaks this lock. By maintaining a large pool of expert weights and activating only a small fraction per token, the architecture decouples the model's total parameter count from its per-token floating-point operations. The experts are the mechanism by which this decoupling is achieved, so understanding their design is central to understanding why MoE works at all.

This chapter examines what experts are, how they are structured, and the architectural decisions that determine their effectiveness. We will see why the feed-forward network became the dominant choice for expert implementation, how to manage the flow of tokens through experts during batch processing, and where to place expert layers within the transformer architecture. We will also examine the parameter-counting arithmetic that makes MoE scaling attractive, work through a numerical example of capacity-constrained routing, and discuss the practical limitations that arise when deploying expert-based models. By the end, you will have a thorough understanding of the building blocks that make sparse expert models function.

Before the implementation details, define what an expert is and its role in the architecture. The term "expert" suggests specialization, and MoE experts often develop distinct functional roles during training. Some experts might specialize in processing mathematical expressions, others in handling conversational text, and still others in domain-specific terminology. This specialization emerges from the routing mechanism rather than being explicitly programmed: gradient descent makes certain experts more effective for particular input types, and the routing mechanism learns to use that structure. No engineer hand-labels which expert should handle which kind of input; the division of labor arises from optimizing the training objective.

An expert is a small neural-network component, typically the size of a standard feed-forward block, rather than a separate model or reasoning system. It receives a token representation, applies a learned non-linear transformation, and returns a representation of the same dimensionality. The expert's output is weighted by the router's confidence score and added to the residual stream, as a standard FFN output would be. To the surrounding transformer, an MoE layer with sparse expert routing resembles a dense FFN layer: a vector goes in and a transformed vector comes out. The difference is that a dense layer uses one shared FFN, whereas an MoE layer uses one of $N$ specialized FFNs.

Experts address a representational as well as a computational problem. A single dense FFN must learn one set of weight matrices that works reasonably well for all possible tokens. That generalist weight set may be adequate for common patterns but suboptimal for rare or specialized ones. An MoE layer can maintain weight sets tuned to different subsets of the token distribution. Each expert can overfit, in a productive sense, to transformations useful for its assigned token types.

An expert is a transformation function that maps an input vector to an output vector within the same space. A token representation therefore emerges transformed but compatible with the rest of the model's architecture:

where:

• $E_i$: the function representing the $i$-th expert
• $\mathbb{R}^d$: the $d$-dimensional real vector space in which token representations live
• $d$: the model's hidden dimension, which determines both input and output size

This dimensionality constraint is not incidental; it is architecturally needed. The expert's output must integrate smoothly with residual connections and subsequent layers in the transformer. If the expert produced vectors of a different size, the residual addition would be impossible. The input and output spaces are identical by design, so the expert slot can substitute for a standard FFN without any other changes to the surrounding block.

For a set of $N$ experts $\{E_1, E_2, \ldots, E_N\}$, each processes its assigned tokens independently. The key insight is that different tokens can be routed to different experts, letting the model to apply specialized processing based on input characteristics. Consider a sentence containing both technical terminology and conversational phrases: the technical terms might be routed to one expert that has learned relevant transformations for such vocabulary, while the conversational phrases flow to a different expert optimized for casual language patterns. This routing happens dynamically for each forward pass, adapting the computation to the specific characteristics of each input.

The most common implementation uses standard feed-forward networks as experts. This choice is not arbitrary; it stems from a basic property of transformer FFN layers. Understanding this property reveals why FFN layers are uniquely suited for expert-based sparsification, and why attention layers, by contrast, are almost never sparsified through expert routing.

Think of the FFN layer in a transformer as a per-token function applier. It takes each token's vector representation, runs it through a two-layer nonlinear network, and returns a transformed vector, completely ignoring all other token positions. This is sometimes called "position-wise" processing, because the same function is applied independently at each position. The attention mechanism, which comes before the FFN in a standard transformer block, handles the cross-position communication: it allows token representations to incorporate information from neighboring positions. Once that communication has occurred, the FFN's job is to transform each resulting representation locally, without any further inter-token interaction.

This position-independence property is exactly what makes FFN layers good candidates for sparsification. If token at position 3 is processed entirely independently of token at position 7 during the FFN phase, then there is no reason these two tokens must use the same FFN weights. We can send token 3 to expert $E_2$ and token 7 to expert $E_5$, and neither computation interferes with the other. The same logic does not apply to attention: because attention computes a weighted sum over all positions simultaneously, you cannot route different positions to different "attention experts" without fundamentally changing the information flow. FFN layers are structurally decoupled from one another across positions, and that decoupling is the key technical reason they become the natural home for MoE routing.

Recall that the feed-forward network in a transformer block operates identically on each position in the sequence, processing each token in isolation. This property, sometimes called "position-wise" or "token-wise" processing, means that the FFN applies the exact same transformation to every token without any awareness of neighboring tokens. The computation for any single token $x$ is self-contained. Mathematically, the FFN applies a point-wise transformation:

We can write this transformation for a single token's vector $x$ as:

where:

• $x$: the input vector corresponding to a specific token position, with dimension $d_{\text{model}}$
• $W_1$: the weight matrix for the first linear transformation, mapping from $\mathbb{R}^{d_{\text{model}}}$ to $\mathbb{R}^{d_{\text{ff}}}$
• $b_1$: the bias vector for the first linear layer, with dimension $d_{\text{ff}}$
• $W_1 \cdot x + b_1$: the projection into the intermediate expansion dimension, before any nonlinearity
• $\sigma$: the non-linear activation function, typically ReLU or GELU, applied element-wise
• $\sigma(W_1 \cdot x + b_1)$: the activated intermediate representation, with each dimension independently passed through the nonlinearity
• $W_2$: the weight matrix for the second linear transformation, mapping from $\mathbb{R}^{d_{\text{ff}}}$ back to $\mathbb{R}^{d_{\text{model}}}$
• $b_2$: the bias vector for the output layer, with dimension $d_{\text{model}}$

Let's trace through this computation to build intuition. The input vector $x$ first undergoes a linear transformation through $W_1$, which projects it from the model dimension $d_{\text{model}}$ into a higher-dimensional intermediate space (typically four times larger). This expansion gives the network room to represent more complex combinations of the input features: in a $d_{\text{model}} = 1024$ model with a standard four-times expansion, the intermediate representation has 4096 dimensions. The bias term $b_1$ shifts this projection, and the activation function $\sigma$ introduces nonlinearity by selectively suppressing or passing through different components of the intermediate representation. The choice of activation matters here: ReLU zeroes out negative values entirely, creating hard sparsity in the intermediate activations, while GELU provides a smoother transition that most modern models prefer. Finally, $W_2$ projects the result back to the original dimensionality, with $b_2$ giving a final bias adjustment.

Why does this formula make sense? Notice that the two-layer structure is the minimal architecture that can represent arbitrary nonlinear functions given enough hidden units, a result dating back to the universal approximation theorem. The expansion-then-contraction shape (wide in the middle, narrow at the edges) is a practical implementation of this principle: the wide middle layer provides the capacity for complex representations, and the contraction back to the model dimension ensures compatibility with the residual stream. The position-independence falls directly out of the fact that $W_1$, $b_1$, $W_2$, and $b_2$ are shared parameters applied identically to each position: nowhere in this formula does the position index appear.

This position-independence is important for MoE. Because each token passes through the FFN independently, we can route different tokens to different FFN-style experts without disrupting the computational flow. If the FFN for token at position 3 has no interaction with the FFN for token at position 7, then there is no reason they must use the same FFN weights. Attention layers, by contrast, require all positions to interact simultaneously through the query-key-value mechanism. This makes them poor candidates for expert-based sparsification.

Each expert in an MoE layer typically mirrors the standard transformer FFN structure, but with its own independent set of parameters. Building on the gated architectures covered in Part XV, most modern expert implementations use the SwiGLU variant. This architecture splits the input into two parallel paths: one for gating and one for features. The gate acts as a dynamic filter that modulates which information from the feature path is allowed to flow through to the output projection. This gating mechanism is inspired by long short-term memory (LSTM) cells, where gates selectively retain or forget information, but applied to the feed-forward context rather than recurrence.

To understand the motivation for gated FFNs over the simpler two-layer variant, consider what happens with ReLU activation. The ReLU activation sets negative intermediate values to zero, effectively performing hard binary gating: a neuron either fully activates or is entirely silent. SwiGLU generalizes this: instead of a fixed zero threshold, the network learns a soft, continuous gate for each dimension based on the current input. This learned gating makes the expert's behavior more adaptive and context-sensitive, which is particularly valuable in the sparse MoE setting where each expert sees only a subset of the data distribution and must be maximally expressive within its assigned domain.

The SwiGLU expert architecture computes its output through the following sequence of operations. The input token representation is projected into a higher-dimensional space along two parallel paths, and their element-wise product forms the pre-output representation:

where:

• $x$: the input token representation vector, with dimension $d_{\text{model}}$
• $W_{\text{gate}}^{(i)}$: the gating projection weights for expert $i$, with shape $(d_{\text{ff}}, d_{\text{model}})$
• $W_{\text{up}}^{(i)}$: the feature up-projection weights for expert $i$, with the same shape $(d_{\text{ff}}, d_{\text{model}})$
• $\text{SiLU}$: the Sigmoid Linear Unit activation function, defined as $\text{SiLU}(z) = z \cdot \sigma(z)$ where $\sigma$ is the sigmoid function
• $\text{Gate}(x)$: the gating mask values computed by the first path, with dimension $d_{\text{ff}}$
• $\text{Features}(x)$: the candidate features computed by the second path, also with dimension $d_{\text{ff}}$
• $\odot$: the element-wise (Hadamard) multiplication operator, which filters features using the gate values
• $W_{\text{down}}^{(i)}$: the down-projection weights for expert $i$, with shape $(d_{\text{model}}, d_{\text{ff}})$, transforming back to the model dimension

To understand why this gated structure is valuable, consider what each component contributes. The up-projection $W_{\text{up}}^{(i)} x$ generates candidate features by expanding the input into a higher-dimensional space where the model can represent more complex patterns. Meanwhile, the gating path $\text{SiLU}(W_{\text{gate}}^{(i)} x)$ computes a vector of values that can range smoothly above and below zero (since SiLU is not bounded like sigmoid), acting as a soft, content-dependent filter. Each element of this gating vector indicates how much the corresponding candidate feature should be emphasized or suppressed.

The element-wise product ($\odot$) then filters the candidates with this gate, letting the network suppress or preserve specific features before the final projection $W_{\text{down}}^{(i)}$. This gating mechanism gives the expert fine-grained, input-dependent control over which information flows through. Some dimensions might be nearly silenced (gate value close to zero) while others pass through strongly (large positive gate value); the decision is dynamic for each input rather than fixed by the weight matrices alone. This selective filtering allows experts to develop specialized behaviors for different input types during training.

Why does this formula make sense? Notice that the two separate projections $W_{\text{gate}}^{(i)}$ and $W_{\text{up}}^{(i)}$ serve entirely different purposes despite having the same shape. The gate path learns to recognize which features in the input are relevant for the current transformation, while the up path learns what the actual feature values should be. The product of these two paths means that only features deemed relevant by the gate path are allowed to influence the output. This is fundamentally more expressive than a simple two-layer FFN where the activation function applies the same threshold rule to every input uniformly.

The key parameters defining an expert's architecture include:

• Input/output dimension ($d_{\text{model}}$): Matches the transformer's hidden dimension, typically 2048 to 8192 in large models. This must be identical for all experts since they all receive the same token representations from the attention layer.
• Intermediate dimension ($d_{\text{ff}}$): The expansion dimension within the expert. For standard FFNs this is often $4d_{\text{model}}$. For SwiGLU variants, a common choice is $\frac{8}{3}d_{\text{model}}$, which preserves the parameter count of the original FFN despite having three weight matrices instead of two (since each of the three matrices is proportionally smaller).
• Activation function: SiLU (also called Swish) for gated variants, or GELU or ReLU for standard two-layer FFNs.

Each expert maintains its own weight matrices, completely independent of other experts. This independence is what enables specialization: expert $E_1$ can learn entirely different transformations than expert $E_2$. During training, the gradient updates for each expert depend only on the tokens routed to that expert, letting different experts to develop distinct computational roles. If expert $E_3$ only ever sees tokens from formal academic writing, its weights will be pushed by gradients shaped entirely by that sub-distribution, naturally developing transformations that work well for that register.

To evaluate the memory footprint of an MoE layer, we sum the parameters across all experts. Understanding this calculation helps you anticipate memory requirements and make informed architectural decisions. The math is straightforward, but tracing through it carefully reveals the core efficiency tradeoff of MoE.

Consider an MoE layer with $N$ experts, where each expert is a standard two-layer FFN with intermediate dimension $d_{\text{ff}}$. For a single expert, the parameter count breaks down into the two weight matrices and their associated bias vectors. We can compute this systematically:

where:

• $N_{\text{expert}}$: the total parameter count for a single expert
• $N_{\text{weights}}$: the parameter count for the weight matrices alone
• $N_{\text{biases}}$: the parameter count for the bias vectors
• $d_{\text{model}}$: the hidden dimension of the model
• $d_{\text{ff}}$: the intermediate dimension inside the expert FFN
• $2 \cdot d_{\text{model}} \cdot d_{\text{ff}}$: parameters for the two weight matrices ($W_1$ and $W_2$), each of size $d_{\text{model}} \times d_{\text{ff}}$
• $d_{\text{ff}} + d_{\text{model}}$: parameters for the bias vectors ($b_1$ of size $d_{\text{ff}}$ and $b_2$ of size $d_{\text{model}}$)

Let's trace through why the weight matrix formula has the factor of 2. The first weight matrix $W_1$ has shape $(d_{\text{ff}}, d_{\text{model}})$, mapping from the model dimension to the intermediate dimension, contributing $d_{\text{model}} \times d_{\text{ff}}$ parameters. The second weight matrix $W_2$ has shape $(d_{\text{model}}, d_{\text{ff}})$, mapping back from the intermediate dimension to the model dimension, contributing another $d_{\text{ff}} \times d_{\text{model}}$ parameters. Together, these two matrices account for the $2 \cdot d_{\text{model}} \cdot d_{\text{ff}}$ term. The bias vectors add a relatively small number of additional parameters: $d_{\text{ff}}$ for $b_1$ and $d_{\text{model}}$ for $b_2$.

For $N$ experts, the total MoE layer parameters are approximately $N \times 2 \times d_{\text{model}} \times d_{\text{ff}}$. A model with 8 experts in each MoE layer has roughly 8 times the FFN parameters of an equivalent dense model. However, if only 2 experts activate per token (the "top-2" routing scheme used by Mixtral), the compute per forward pass remains comparable to a dense model with 2 times the FFN size, which is still much less than 8 times. This is the basic tradeoff of MoE: we store many more parameters than we use for any single input, gaining model capacity without proportional computational cost. The price we pay is in memory, not arithmetic.

When processing batches of sequences, MoE models face a unique challenge that dense transformers never encounter: different tokens route to different experts, but hardware prefers uniform workloads. GPUs and TPUs are most efficient when performing the same operation on identically-sized blocks of data. Batch matrix multiplications, in particular, assume that every element of the batch has the same shape and that no element requires extra synchronization. MoE routing violates this assumption by design: in a perfectly-functioning model, popular experts attract many tokens while niche experts process only a few. Without some mechanism to manage this variability, the hardware efficiency that motivated MoE in the first place can be partially or entirely lost to irregular computation patterns.

Think of expert capacity as a physical queue in front of each expert's desk in our library analogy. Each queue can hold at most $K$ patrons at a time. When a popular expert's queue is full and a new patron arrives, that patron cannot wait and instead must either go to their second-best option or handle their query on their own using only the general information they already have. This queue limit prevents the popular expert from becoming a bottleneck that makes everyone else wait, at the cost of occasional suboptimal service for tokens that get redirected or dropped. The capacity factor $C$ determines how long each queue can be relative to the perfectly-balanced case.

Expert capacity mechanisms manage this tension by imposing limits on how many tokens each expert can process per batch. The limits are not arbitrary restrictions: they are engineering constraints that make it possible to implement MoE layers as regular tensor operations on hardware that demands predictable memory layouts and computation graphs. Without these limits, implementing an MoE layer would require dynamic tensor shapes that change based on the routing decisions, making the computation far harder to optimize and parallelize.

The key insight is that capacity constraints create a tradeoff: tighter constraints improve hardware utilization and reduce memory overhead, but they also mean some tokens receive inferior processing (or none at all) when routing is imbalanced. Looser constraints preserve routing quality but waste memory and compute on empty buffer slots. Finding the right capacity factor is therefore a hyperparameter tuning problem with direct implications for both model quality and training efficiency.

Consider a batch of 1024 tokens being processed by an 8-expert MoE layer. If routing were perfectly balanced, each expert would receive exactly 128 tokens. In practice, some tokens cluster around certain experts while others receive few tokens. Without constraints, popular experts become computational bottlenecks while unpopular experts sit idle.

This load imbalance creates practical problems for efficient computation. Modern hardware, particularly GPUs, achieves peak throughput when performing regular, predictable operations on uniformly-sized data. When one expert must process 300 tokens while another processes only 50, we cannot fully utilize parallel processing capabilities. The expert with 300 tokens becomes a serialization point, and the expert with 50 tokens wastes potential compute cycles.

Load imbalance also creates memory management difficulties. In a static-graph compilation framework (such as XLA on TPU), every tensor must have a known shape at compile time. If expert queue lengths vary dynamically based on routing decisions, the compiler cannot determine tensor shapes in advance. The capacity mechanism solves this by pre-allocating a fixed-size buffer for each expert: we know at compile time that each expert will process at most $K$ tokens, so we can pre-allocate tensors of exactly that size. This static shape guarantee is what makes MoE layers compilable to efficient hardware kernels. The capacity constraint therefore is a load-balancing heuristic and a basic requirement for static-graph compilation.

To prevent bottlenecks where popular experts become overloaded, we calculate the expert capacity $K$ (the maximum tokens per expert) based on the total batch size and a buffer multiplier. This approach provides a principled way to balance routing flexibility against computational efficiency. We want each expert's buffer to be large enough to absorb typical routing variance without being so large that most of the allocated space goes unused.

The capacity formula computes how many tokens each expert is allowed to hold:

where:

• $K$: the expert capacity (maximum tokens per expert for this batch)
• $C$: the capacity factor ($\geq 1.0$), a buffer multiplier that determines headroom above the perfectly-balanced allocation
• $T$: the total number of tokens in the batch
• $E$: the total count of available experts
• $T/E$: the baseline load per expert under perfectly balanced routing, serving as the reference point

The intuition behind this formula is straightforward. The ratio $T/E$ represents the average number of tokens each expert would receive under perfectly balanced routing. In our example with 1024 tokens and 8 experts, this baseline is 128 tokens per expert. The capacity factor $C$ then scales this baseline to determine how much deviation from perfect balance we allow. Setting $C = 1.25$ means each expert can accept up to 160 tokens (128 times 1.25). This provides a 25% buffer above the perfectly balanced load.

A capacity factor of 1.0 means each expert can handle exactly its "fair share" of tokens. Values greater than 1.0 provide headroom for imbalanced routing. The choice of $C$ affects model behavior in several ways:

• $C = 1.0$: Minimal buffer. Routing must be nearly perfectly balanced to avoid token dropping. Useful when load balancing losses are strong and routing is well-behaved.
• $C = 1.25$: Common choice in practice. This provides a 25% buffer for routing variance. This value was used in the original Switch Transformer paper and has become something of a default.
• $C = 2.0$: Large buffer. Rarely causes overflow but doubles the per-expert memory allocation. Useful for smaller models where routing is more difficult to balance.

Choosing the right capacity factor involves balancing memory efficiency against routing flexibility. A higher capacity factor allocates more buffer space for each expert's token queue, consuming more memory. However, it also reduces the probability that tokens are dropped due to capacity overflow, improving model quality. Empirically, the sweet spot for most models is between 1.0 and 1.5, with 1.25 being a reasonable starting point.

When an expert reaches capacity, additional tokens routed to it are typically "dropped": they bypass the MoE layer entirely and pass through via the residual connection. This ensures forward passes complete in a predictable amount of time, but it means some tokens receive less processing than intended. The model still produces an output for every dropped token, but that output is simply the unchanged input representation, as if the MoE layer were not there at all for those positions.

The drop rate provides a diagnostic signal during training. If 10% of tokens are dropped during training, the routing mechanism is not distributing load effectively, motivating the auxiliary losses we will explore in upcoming chapters on load balancing. You monitor drop rates as a health metric for your MoE training runs, as sustained high drop rates often correlate with degraded downstream performance. A good rule of thumb is that drop rates above 1 to 2% are worth investigating, and drop rates above 5% indicate a routing problem that should be addressed rather than tolerated.

Small amounts of token dropping are not necessarily harmful. Because the routing mechanism is differentiable (the router learns from gradients flowing back through the routing weights), moderate overflow can guide the router toward more balanced distributions. The prospect of dropped tokens creates an implicit incentive for the router to spread load more evenly, complementing explicit load-balancing loss terms. Consistently high drop rates indicate that the model is failing to use its expert capacity, and performance suffers.

Training typically uses fixed capacity factors to enable efficient batched operations. The predictable memory allocation allows for optimized kernel implementations and stable training dynamics. Fixed capacity also makes gradient computation more predictable, since the set of tokens that receive expert processing is determined before the forward pass rather than dynamically during it.

During inference, especially for single sequences, capacity constraints may be relaxed. Some implementations process all routed tokens regardless of capacity during inference, accepting variable compute time for better quality. This flexibility makes sense because inference often involves smaller batches where strict capacity constraints provide less benefit, and you typically prefer accuracy over strict latency guarantees. Single-request inference, for example, processes only one sequence at a time, so there is no competing workload to balance against and no risk of one expert starving another. For these cases, you can simply route every token to its top-k experts and process them all, recovering the full quality of the model.

Some production deployment strategies use a hybrid approach: during inference, maintain capacity constraints but set the capacity factor higher than during training. This preserves the memory and latency predictability of capacity-constrained routing while letting more routing flexibility than the training configuration. The optimal inference capacity factor is something you tune based on your latency budget and quality requirements.

Choosing the number of experts involves balancing specialization potential, memory requirements, and computational efficiency. Different models have explored configurations ranging from 8 to thousands of experts, and there is no universally optimal number. The right choice depends on the training data distribution, available hardware, and target use case. Understanding the tradeoffs qualitatively helps you reason about architectural decisions in the models you read about and build.

Think of expert count selection as deciding how many specialized workshops to maintain in a factory. A factory with 4 specialized workshops can route different types of work to appropriate stations, but the specialization is relatively coarse. A factory with 64 workshops can be much more finely specialized, with each workshop handling a narrow type of task extremely well. But maintaining 64 workshops requires more floor space, more staff, and more coordination overhead. At some point, adding more workshops provides diminishing returns if there is not enough work of each specialized type to keep every workshop productive. The optimal number of workshops depends on how diverse the incoming work is and how much floor space (memory) you have available.

The choice of expert count is also intimately connected to the routing topology. A model with 8 experts and top-2 routing activates 25% of its experts per token. A model with 64 experts and top-2 routing activates only about 3% of its experts per token. The second model has much greater sparsity, which means each expert sees a much smaller fraction of the training distribution. This can lead to richer specialization but also to undertraining risks if some experts see too few tokens to learn effective transformations. The relationship between expert count, routing sparsity, and training dynamics is an active area of research, and the empirical findings from major MoE papers provide useful reference points.

Increasing the number of experts has several competing effects that you need to weigh against each other. More experts generally improves the model's capacity to specialize, which can improve quality on diverse tasks. But it also increases the engineering and systems complexity of training and deployment.

Benefits of more experts:

• Greater total parameter count without proportional compute increase, giving the model more capacity to represent complex functions
• Finer-grained specialization potential, letting experts to develop highly targeted transformations for narrow sub-distributions
• Better scaling on distributed systems with expert parallelism, since more experts can be spread across more devices

Costs of more experts:

• More memory required to store all expert weights, even those not activated for a given token
• Communication overhead when experts span multiple devices, requiring tokens to be transferred to the device holding the relevant expert
• Risk of some experts being chronically underutilized, leading to undertrained weights that hurt quality when they are eventually selected

The communication overhead point deserves special attention because it is often the binding constraint in practice. In a model where each expert lives on a separate device (the "expert parallelism" strategy), every routing decision that sends a token to an expert on a different device requires network communication. With 8 experts on 8 devices, routing is relatively simple: most tokens stay on their local device or transfer to one of a small number of remote devices. With 1024 experts on 64 devices, routing becomes a complex all-to-all communication problem. The time spent on this communication can easily dominate the time spent on actual computation, eliminating the efficiency gains that motivated MoE in the first place.

Major MoE models have converged on different expert counts based on their design goals and available hardware:

Switch Transformer demonstrated that even single-expert routing (top-1, meaning only one active expert per token) works well when the expert count is very large, letting massive parameter counts while keeping per-token compute low. The original Switch Transformer scaled to over a trillion parameters by using thousands of experts per layer. Mixtral showed that a much simpler configuration (8 experts, top-2 routing) achieves strong quality while remaining practical to deploy on commodity hardware. DeepSeek-MoE explored a middle ground, using many fine-grained experts (64) with a higher active count (6) to capture more routing granularity.

More experts with fewer active per token increases the capacity-to-compute ratio. A model with 64 experts but only 2 active has 32 times more parameters than its per-token compute would suggest. This ratio affects:

• Memory requirements: all expert weights must be stored in memory (or on the same device in a distributed setting), even those that will not be used for any given batch
• Potential for overfitting: more capacity than the training data can support leads to experts that memorize training patterns rather than learning generalizable transformations
• Inference efficiency: expert weights may need to be loaded from slower memory (system RAM or disk) if the GPU memory cannot hold all of them simultaneously

The key insight here is that the right capacity-to-compute ratio depends on the training data volume as much as the hardware configuration. A model trained on 1 trillion tokens can productively use much higher capacity than a model trained on 100 billion tokens, because there is sufficient diversity in the data to train every expert to do something useful. Scaling laws for MoE are an active research area, and the relationship between expert count, training tokens, and final quality is not yet as well-characterized as for dense models.

Some architectures include "shared" experts that process every token alongside the sparse routing to specialized experts. DeepSeek-MoE uses this approach, with 2 shared experts plus routing among 64 specialized experts. Shared experts capture common transformations that benefit every token, such as basic syntactic processing or common lexical patterns, while specialized experts handle token-specific or domain-specific processing.

The intuition for shared experts comes from the observation that not all processing is specialized. Every sentence, regardless of its domain, involves basic grammatical relationships, function-word handling, and common semantic operations that benefit from consistent treatment. If specialized experts learn these common operations through repeated routing of common tokens, they may not have sufficient capacity left for specialization. Dedicated shared experts can handle these universal computations, leaving specialized experts to focus on differences between domains and token types.

Not every layer in a transformer needs to be an MoE layer. The placement of expert layers significantly impacts model behavior and efficiency. This section examines the design patterns that have proven effective in practice and explains the reasoning behind the architectural choices that major models have converged on.

The transformer architecture has a regular repeating structure: each layer consists of a self-attention module followed by a feed-forward network, with layer normalization and residual connections wrapping each component. When introducing MoE, the key question is which components to replace with expert versions and in which layers. The answer is not purely a matter of taste: it reflects deep considerations about what each component does and whether sparse routing is compatible with that function.

Think of the choice of where to place MoE layers as deciding which departments in a company should have specialists and which should have generalists. In most companies, the legal and finance departments are highly specialized (only certain people can do that work), while the IT helpdesk and facilities management are shared across the whole company. Similarly, certain layers of a transformer may benefit from specialization while others are better off shared. The empirical pattern that has emerged is that the FFN component of the transformer block is the right place for expert routing, and that alternating between dense and MoE layers strikes a useful balance.

The placement decision also has consequences for the routing mechanism. If MoE layers appear at every depth, the router at each layer must learn to distribute tokens appropriately for that layer's function. Early layers tend to handle lower-level linguistic structure (morphology, basic syntax), while deeper layers handle higher-level semantics and task-specific reasoning. A model where every layer is MoE must learn routing policies that account for this depth-dependent specialization. Alternating patterns allow dense layers to create shared representations that subsequently MoE layers can then specialize from, potentially making the routing problem simpler.

The standard approach replaces some or all FFN layers with MoE layers while keeping attention layers unchanged. This follows naturally from the position-independence property discussed earlier.

In a transformer block, the computation flows through a sequence of operations. The input first passes through layer normalization, then through the attention mechanism where tokens exchange information. After a residual connection adds the input back to the attention output, another layer normalization prepares the representation for the FFN (or MoE) layer. Finally, another residual connection completes the block. The FFN position, coming after attention has already enabled cross-position communication, becomes the natural location for the MoE layer. At this point in the computation, each token has already had the opportunity to incorporate information from its context, and the remaining job is to transform that enriched representation in a token-specific way.

The FFN position becomes the MoE position. Attention layers remain dense. This keeps all positions can still interact before expert-specialized processing. This design principle reflects the distinct roles of attention and FFN layers in transformers: attention handles information routing between positions, while FFN layers turn the representation at each position independently. MoE exploits this independence by running different transformations for different tokens, all without interfering with the cross-position communication that attention handles.

Several patterns have been explored for alternating dense and MoE layers, each which reflects different assumptions about how much expert processing is beneficial relative to the overhead it introduces:

• Every layer MoE: All FFN layers become MoE layers. This maximizes expert utilization but increases routing overhead and may not improve quality proportionally. Training is also more complex, since every layer introduces the routing instability and load-balancing challenges associated with MoE.

• Alternating (every 2nd layer): A common pattern in practice. Every other FFN layer is replaced with MoE. Mixtral uses this approach. Dense layers provide regular checkpoints of shared processing between expert layers, which may help stabilize the representation and prevent expert drift across layers.

• Periodic (every nth layer): More aggressive interleaving, such as MoE at every 4th layer. This reduces router overhead significantly but limits expert specialization opportunities, since tokens pass through fewer expert layers overall.

Some architectures vary the number of experts by layer depth. Early layers might use fewer experts because input patterns are more uniform and syntax-driven, while later layers use more experts where task-specific and domain-specific specialization becomes most valuable. This pattern appears in certain custom architectures, though major publicly released models typically use uniform expert counts for simplicity. Token features become more diverse in deeper layers, so later layers may gain more from fine-grained specialization than early layers.

An emerging technique shares expert weights across multiple layers while maintaining layer-specific routers. This reduces total parameters while preserving adaptive routing. Each layer can route differently even if the underlying expert computations are shared. The key insight is that the specialization in MoE comes partly from routing decisions (which expert gets which token) and partly from the expert's weight matrices (what transformation that expert applies). Cross-layer sharing retains the routing flexibility while reducing the parameter count by reusing expert weights. This is an active area of research, and we may see more production models adopt it as the field develops better understanding of how routing and weight specialization interact.

Let's work through a concrete numerical trace of how an MoE layer processes a small batch of tokens. This example makes the abstract architecture tangible and helps you see exactly where the parameter count, capacity limits, and routing decisions come together.

Suppose we have a small MoE layer with the following configuration: $d_{\text{model}} = 4$, $d_{\text{ff}} = 8$, 4 experts, and top-1 routing. We have a batch of 8 tokens, and the routing scores (which we assume have already been computed by the router) assign each token to one expert:

• Tokens 0, 2, 4: assigned to Expert 0 (3 tokens)
• Tokens 1, 5: assigned to Expert 1 (2 tokens)
• Token 3: assigned to Expert 2 (1 token)
• Tokens 6, 7: assigned to Expert 3 (2 tokens)

With a capacity factor of $C = 1.25$ and 4 experts over 8 tokens, the capacity per expert is $K = 1.25 \times (8/4) = 2.5$, which rounds down to 2. Expert 0 is assigned 3 tokens but has capacity for only 2. Token 4, which has the lowest router confidence score among Expert 0's three assigned tokens, is dropped. It will pass through the MoE layer unchanged via the residual connection.

Now let's trace the computation for token 0, which is routed to Expert 0. Token 0's hidden representation is a 4-dimensional vector, let's call it $x_0 = [0.5, -0.3, 0.8, 0.2]$. Expert 0 applies its weight matrices:

First, the up-projection: $h = W_1 x_0 + b_1$, creating an 8-dimensional intermediate vector.

Then, the activation: $a = \sigma(h)$, where $\sigma$ applies SiLU element-wise.

Finally, the down-projection: $y_0 = W_2 a + b_2$, creating the 4-dimensional output.

This output $y_0$ is then scaled by the router's confidence weight for Expert 0 (let's say 0.87) and added to the residual: the final output for token 0 is $x_0 + 0.87 \cdot y_0$.

Token 4, the dropped token, skips the expert computation entirely. Its output is simply $x_4 + 0 = x_4$: the residual connection carries the unchanged representation to the next layer. This means the MoE layer contributes nothing to token 4's representation, which is the cost of token dropping.

The key observation from this trace is that each expert processes its assigned tokens independently and in parallel. Expert 0 processes tokens 0 and 2 simultaneously (as a batch of size 2). Expert 1 processes tokens 1 and 5. Expert 2 processes token 3. Expert 3 processes tokens 6 and 7. All four experts run at the same time on different hardware units. The total wall-clock time for this MoE layer is determined by the slowest expert, which in this case is Expert 0 (processing 2 tokens) rather than Expert 2 (processing 1 token). This illustrates why load balancing matters: uneven load means some hardware is busy while other hardware waits, reducing efficiency.

Let's implement a simple expert network structure and explore capacity-constrained processing.

Understanding what each hyperparameter controls helps you reason about MoE configurations you encounter in the literature or need to tune yourself. The key parameters for the Expert implementation are:

• d_model: The dimensionality of input and output tokens. This must match the transformer's hidden size since experts sit inside the transformer block and their outputs flow into residual connections. Changing this value requires changing the entire model, including the expert.
• d_ff: The intermediate expansion dimension within each expert FFN. This controls the representational capacity of each expert. Larger values mean more expressive experts but more memory and compute per activated expert. The standard is 4 times d_model for two-layer FFNs and approximately 2.67 times d_model for SwiGLU variants.
• num_experts: The total number of experts available in the layer. This is the primary lever for trading memory against specialization. More experts means more total parameters and potentially finer-grained specialization, but also more memory and communication overhead.
• capacity_factor: A multiplier determining the maximum number of tokens an expert can process relative to a perfectly balanced load. This is the primary lever for trading routing quality against hardware efficiency. Higher values reduce token dropping but increase memory allocation per expert.

Expert networks introduce several practical challenges. The most immediate is memory overhead: storing $N$ experts requires $N$ times the memory of a single FFN layer, even though most experts remain idle for any given token. A Mixtral 8x7B model stores 8 separate FFN copies per MoE layer, consuming significant GPU memory despite only activating 2 at inference time. This memory-compute asymmetry means MoE models often require more hardware to deploy than their per-token FLOP count suggests. A model computationally equivalent to a 7B dense model at inference time may require the memory bandwidth and storage of a 47B parameter model because all expert weights must be resident in memory.

This memory-compute split creates practical deployment challenges that do not arise with dense models. For dense models, memory and compute scale together: a model that needs more memory also needs more compute, and hardware allocation is relatively straightforward. For MoE models, you must separately optimize for peak memory (which is determined by all expert weights) and throughput (which is determined by active expert weights). Quantizing expert weights to lower precision (int4 or int8) is one common approach to managing this tradeoff, since quantization reduces memory footprint with relatively modest quality degradation for the inactive experts.

Expert underutilization presents another concern. Without careful load balancing, some experts may receive far fewer tokens during training, leading to undertrained weights that hurt model quality when they are eventually selected. This creates a feedback loop: underutilized experts produce worse outputs, causing the router to select them even less frequently. The auxiliary losses we will discuss in upcoming chapters directly address this problem by penalizing routing distributions that are too concentrated. The severity of this collapse behavior depends on the routing mechanism and the training hyperparameters, and it remains one of the more delicate aspects of MoE training.

The capacity mechanism itself introduces discontinuities in the training signal. Tokens dropped due to capacity overflow still contribute to the loss but receive no expert processing, creating a mismatch between forward computation and gradient flow. The router does not receive a gradient signal telling it that certain tokens were dropped: from the perspective of the loss function, those tokens simply passed through the residual connection, which is a valid (if suboptimal) outcome. Various techniques attempt to make this process more differentiable, but some information loss is inherent to hard capacity constraints. This is one reason why some recent MoE variants have explored soft capacity mechanisms and token merging approaches that avoid hard dropping entirely.

Load balancing at inference time is also a non-trivial concern. During training, the model sees a diverse mixture of inputs, and the routing distribution averages out over many batches. At inference time, however, you may process a stream of requests that are all from the same domain: a coding assistant receiving only code inputs, for example, may route almost all tokens to a small subset of experts. This request-time imbalance can cause expert parallelism bottlenecks that did not appear during training, degrading inference throughput. Some deployment systems address this by routing similar requests together to avoid within-batch imbalance, but this adds infrastructure complexity.

Finally, expert placement decisions remain somewhat empirical. While alternating dense and MoE layers has become standard, the optimal interleaving pattern likely varies by task and scale. Some recent work suggests that certain layer positions benefit more from expert diversity than others, but clear design principles are still emerging from the research community. The field does not yet have the equivalent of scaling laws for MoE placement: we know that some configurations work well empirically, but a principled theory of why certain placements outperform others is still an open question.

Expert networks form the computational substrate of Mixture of Experts models. The key insights from this chapter include:

• Experts are independent FFNs that mirror standard transformer feed-forward layers. The position-independence of FFN operations makes them natural candidates for sparse expert-based routing, while attention layers are poor candidates because they require cross-position communication that cannot be easily distributed across independent subnetworks.

• The SwiGLU architecture used in modern expert implementations adds a learned, input-dependent gating mechanism that allows experts to selectively filter their intermediate representations. This makes experts more expressive than simple two-layer FFNs and better suited to developing specialized transformations for narrow input distributions.

• Expert capacity manages the tension between flexible routing and efficient batched computation. The capacity factor determines how much routing imbalance the system tolerates before dropping tokens. The right capacity factor balances memory efficiency against routing quality, and monitoring drop rates during training is needed for diagnosing routing health.

• Expert count involves tradeoffs between specialization potential, memory requirements, and utilization efficiency. Successful models range from 8 experts (Mixtral) to thousands (Switch Transformer). More experts means more capacity but also more memory, more communication overhead in distributed settings, and greater risk of expert underutilization.

• Expert placement typically replaces FFN layers while keeping attention layers dense. Alternating dense and MoE layers is a common pattern that balances routing overhead with expert specialization, and this approach has been validated empirically across multiple major MoE architectures.

• Practical limitations of expert networks include memory-compute asymmetry, expert collapse (underutilization feedback loops), non-differentiable capacity constraints, and inference-time load imbalance. These challenges motivate the auxiliary losses and deployment engineering techniques covered in subsequent chapters.

With the expert architecture established, the natural question becomes: how does the model decide which expert should process each token? The next chapter on gating networks addresses exactly this question, introducing the routing mechanisms that make sparse expert selection possible and the loss functions that keep routing balanced during training.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about expert networks in Mixture of Experts architectures.