Expert Networks › Experts as Feed-Forward Networks

The most common implementation uses standard feed-forward networks as experts. This choice is not arbitrary; it stems from a basic property of transformer FFN layers. Understanding this property reveals why FFN layers are uniquely suited for expert-based sparsification, and why attention layers, by contrast, are almost never sparsified through expert routing.

Think of the FFN layer in a transformer as a per-token function applier. It takes each token's vector representation, runs it through a two-layer nonlinear network, and returns a transformed vector, completely ignoring all other token positions. This is sometimes called "position-wise" processing, because the same function is applied independently at each position. The attention mechanism, which comes before the FFN in a standard transformer block, handles the cross-position communication: it allows token representations to incorporate information from neighboring positions. Once that communication has occurred, the FFN's job is to transform each resulting representation locally, without any further inter-token interaction.

This position-independence property is exactly what makes FFN layers good candidates for sparsification. If token at position 3 is processed entirely independently of token at position 7 during the FFN phase, then there is no reason these two tokens must use the same FFN weights. We can send token 3 to expert $E_2$ and token 7 to expert $E_5$, and neither computation interferes with the other. The same logic does not apply to attention: because attention computes a weighted sum over all positions simultaneously, you cannot route different positions to different "attention experts" without fundamentally changing the information flow. FFN layers are structurally decoupled from one another across positions, and that decoupling is the key technical reason they become the natural home for MoE routing.

Recall that the feed-forward network in a transformer block operates identically on each position in the sequence, processing each token in isolation. This property, sometimes called "position-wise" or "token-wise" processing, means that the FFN applies the exact same transformation to every token without any awareness of neighboring tokens. The computation for any single token $x$ is self-contained. Mathematically, the FFN applies a point-wise transformation:

We can write this transformation for a single token's vector $x$ as:

$$\text{FFN}(x) = W_2 \cdot \sigma(W_1 \cdot x + b_1) + b_2$$

where:

• $x$: the input vector corresponding to a specific token position, with dimension $d_{\text{model}}$
• $W_1$: the weight matrix for the first linear transformation, mapping from $\mathbb{R}^{d_{\text{model}}}$ to $\mathbb{R}^{d_{\text{ff}}}$
• $b_1$: the bias vector for the first linear layer, with dimension $d_{\text{ff}}$
• $W_1 \cdot x + b_1$: the projection into the intermediate expansion dimension, before any nonlinearity
• $\sigma$: the non-linear activation function, typically ReLU or GELU, applied element-wise
• $\sigma(W_1 \cdot x + b_1)$: the activated intermediate representation, with each dimension independently passed through the nonlinearity
• $W_2$: the weight matrix for the second linear transformation, mapping from $\mathbb{R}^{d_{\text{ff}}}$ back to $\mathbb{R}^{d_{\text{model}}}$
• $b_2$: the bias vector for the output layer, with dimension $d_{\text{model}}$

Let's trace through this computation to build intuition. The input vector $x$ first undergoes a linear transformation through $W_1$, which projects it from the model dimension $d_{\text{model}}$ into a higher-dimensional intermediate space (typically four times larger). This expansion gives the network room to represent more complex combinations of the input features: in a $d_{\text{model}} = 1024$ model with a standard four-times expansion, the intermediate representation has 4096 dimensions. The bias term $b_1$ shifts this projection, and the activation function $\sigma$ introduces nonlinearity by selectively suppressing or passing through different components of the intermediate representation. The choice of activation matters here: ReLU zeroes out negative values entirely, creating hard sparsity in the intermediate activations, while GELU provides a smoother transition that most modern models prefer. Finally, $W_2$ projects the result back to the original dimensionality, with $b_2$ giving a final bias adjustment.

Why does this formula make sense? Notice that the two-layer structure is the minimal architecture that can represent arbitrary nonlinear functions given enough hidden units, a result dating back to the universal approximation theorem. The expansion-then-contraction shape (wide in the middle, narrow at the edges) is a practical implementation of this principle: the wide middle layer provides the capacity for complex representations, and the contraction back to the model dimension ensures compatibility with the residual stream. The position-independence falls directly out of the fact that $W_1$, $b_1$, $W_2$, and $b_2$ are shared parameters applied identically to each position: nowhere in this formula does the position index appear.

This position-independence is important for MoE. Because each token passes through the FFN independently, we can route different tokens to different FFN-style experts without disrupting the computational flow. If the FFN for token at position 3 has no interaction with the FFN for token at position 7, then there is no reason they must use the same FFN weights. Attention layers, by contrast, require all positions to interact simultaneously through the query-key-value mechanism. This makes them poor candidates for expert-based sparsification.

Each expert in an MoE layer typically mirrors the standard transformer FFN structure, but with its own independent set of parameters. Building on the gated architectures covered in Part XV, most modern expert implementations use the SwiGLU variant. This architecture splits the input into two parallel paths: one for gating and one for features. The gate acts as a dynamic filter that modulates which information from the feature path is allowed to flow through to the output projection. This gating mechanism is inspired by long short-term memory (LSTM) cells, where gates selectively retain or forget information, but applied to the feed-forward context rather than recurrence.

To understand the motivation for gated FFNs over the simpler two-layer variant, consider what happens with ReLU activation. The ReLU activation sets negative intermediate values to zero, effectively performing hard binary gating: a neuron either fully activates or is entirely silent. SwiGLU generalizes this: instead of a fixed zero threshold, the network learns a soft, continuous gate for each dimension based on the current input. This learned gating makes the expert's behavior more adaptive and context-sensitive, which is particularly valuable in the sparse MoE setting where each expert sees only a subset of the data distribution and must be maximally expressive within its assigned domain.

The SwiGLU expert architecture computes its output through the following sequence of operations. The input token representation is projected into a higher-dimensional space along two parallel paths, and their element-wise product forms the pre-output representation:

$$\begin{aligned} \text{Gate}(x) &= \text{SiLU}(W_{\text{gate}}^{(i)} \cdot x) \\ \text{Features}(x) &= W_{\text{up}}^{(i)} \cdot x \\ E_i(x) &= W_{\text{down}}^{(i)} \cdot (\text{Gate}(x) \odot \text{Features}(x)) \end{aligned}$$

where:

• $x$: the input token representation vector, with dimension $d_{\text{model}}$
• $W_{\text{gate}}^{(i)}$: the gating projection weights for expert $i$, with shape $(d_{\text{ff}}, d_{\text{model}})$
• $W_{\text{up}}^{(i)}$: the feature up-projection weights for expert $i$, with the same shape $(d_{\text{ff}}, d_{\text{model}})$
• $\text{SiLU}$: the Sigmoid Linear Unit activation function, defined as $\text{SiLU}(z) = z \cdot \sigma(z)$ where $\sigma$ is the sigmoid function
• $\text{Gate}(x)$: the gating mask values computed by the first path, with dimension $d_{\text{ff}}$
• $\text{Features}(x)$: the candidate features computed by the second path, also with dimension $d_{\text{ff}}$
• $\odot$: the element-wise (Hadamard) multiplication operator, which filters features using the gate values
• $W_{\text{down}}^{(i)}$: the down-projection weights for expert $i$, with shape $(d_{\text{model}}, d_{\text{ff}})$, transforming back to the model dimension

To understand why this gated structure is valuable, consider what each component contributes. The up-projection $W_{\text{up}}^{(i)} x$ generates candidate features by expanding the input into a higher-dimensional space where the model can represent more complex patterns. Meanwhile, the gating path $\text{SiLU}(W_{\text{gate}}^{(i)} x)$ computes a vector of values that can range smoothly above and below zero (since SiLU is not bounded like sigmoid), acting as a soft, content-dependent filter. Each element of this gating vector indicates how much the corresponding candidate feature should be emphasized or suppressed.

The element-wise product ($\odot$) then filters the candidates with this gate, letting the network suppress or preserve specific features before the final projection $W_{\text{down}}^{(i)}$. This gating mechanism gives the expert fine-grained, input-dependent control over which information flows through. Some dimensions might be nearly silenced (gate value close to zero) while others pass through strongly (large positive gate value); the decision is dynamic for each input rather than fixed by the weight matrices alone. This selective filtering allows experts to develop specialized behaviors for different input types during training.

Why does this formula make sense? Notice that the two separate projections $W_{\text{gate}}^{(i)}$ and $W_{\text{up}}^{(i)}$ serve entirely different purposes despite having the same shape. The gate path learns to recognize which features in the input are relevant for the current transformation, while the up path learns what the actual feature values should be. The product of these two paths means that only features deemed relevant by the gate path are allowed to influence the output. This is fundamentally more expressive than a simple two-layer FFN where the activation function applies the same threshold rule to every input uniformly.

The key parameters defining an expert's architecture include:

• Input/output dimension ($d_{\text{model}}$): Matches the transformer's hidden dimension, typically 2048 to 8192 in large models. This must be identical for all experts since they all receive the same token representations from the attention layer.
• Intermediate dimension ($d_{\text{ff}}$): The expansion dimension within the expert. For standard FFNs this is often $4d_{\text{model}}$. For SwiGLU variants, a common choice is $\frac{8}{3}d_{\text{model}}$, which preserves the parameter count of the original FFN despite having three weight matrices instead of two (since each of the three matrices is proportionally smaller).
• Activation function: SiLU (also called Swish) for gated variants, or GELU or ReLU for standard two-layer FFNs.

Each expert maintains its own weight matrices, completely independent of other experts. This independence is what enables specialization: expert $E_1$ can learn entirely different transformations than expert $E_2$. During training, the gradient updates for each expert depend only on the tokens routed to that expert, letting different experts to develop distinct computational roles. If expert $E_3$ only ever sees tokens from formal academic writing, its weights will be pushed by gradients shaped entirely by that sub-distribution, naturally developing transformations that work well for that register.

To evaluate the memory footprint of an MoE layer, we sum the parameters across all experts. Understanding this calculation helps you anticipate memory requirements and make informed architectural decisions. The math is straightforward, but tracing through it carefully reveals the core efficiency tradeoff of MoE.

Consider an MoE layer with $N$ experts, where each expert is a standard two-layer FFN with intermediate dimension $d_{\text{ff}}$. For a single expert, the parameter count breaks down into the two weight matrices and their associated bias vectors. We can compute this systematically:

$$\begin{aligned} N_{\text{weights}} &= 2 \cdot d_{\text{model}} \cdot d_{\text{ff}} \\ N_{\text{biases}} &= d_{\text{ff}} + d_{\text{model}} \\ N_{\text{expert}} &= N_{\text{weights}} + N_{\text{biases}} \\ &= 2 \cdot d_{\text{model}} \cdot d_{\text{ff}} + d_{\text{ff}} + d_{\text{model}} \end{aligned}$$

where:

• $N_{\text{expert}}$: the total parameter count for a single expert
• $N_{\text{weights}}$: the parameter count for the weight matrices alone
• $N_{\text{biases}}$: the parameter count for the bias vectors
• $d_{\text{model}}$: the hidden dimension of the model
• $d_{\text{ff}}$: the intermediate dimension inside the expert FFN
• $2 \cdot d_{\text{model}} \cdot d_{\text{ff}}$: parameters for the two weight matrices ($W_1$ and $W_2$), each of size $d_{\text{model}} \times d_{\text{ff}}$
• $d_{\text{ff}} + d_{\text{model}}$: parameters for the bias vectors ($b_1$ of size $d_{\text{ff}}$ and $b_2$ of size $d_{\text{model}}$)

Let's trace through why the weight matrix formula has the factor of 2. The first weight matrix $W_1$ has shape $(d_{\text{ff}}, d_{\text{model}})$, mapping from the model dimension to the intermediate dimension, contributing $d_{\text{model}} \times d_{\text{ff}}$ parameters. The second weight matrix $W_2$ has shape $(d_{\text{model}}, d_{\text{ff}})$, mapping back from the intermediate dimension to the model dimension, contributing another $d_{\text{ff}} \times d_{\text{model}}$ parameters. Together, these two matrices account for the $2 \cdot d_{\text{model}} \cdot d_{\text{ff}}$ term. The bias vectors add a relatively small number of additional parameters: $d_{\text{ff}}$ for $b_1$ and $d_{\text{model}}$ for $b_2$.

For $N$ experts, the total MoE layer parameters are approximately $N \times 2 \times d_{\text{model}} \times d_{\text{ff}}$. A model with 8 experts in each MoE layer has roughly 8 times the FFN parameters of an equivalent dense model. However, if only 2 experts activate per token (the "top-2" routing scheme used by Mixtral), the compute per forward pass remains comparable to a dense model with 2 times the FFN size, which is still much less than 8 times. This is the basic tradeoff of MoE: we store many more parameters than we use for any single input, gaining model capacity without proportional computational cost. The price we pay is in memory, not arithmetic.