Expert Networks › Worked Example: Expert Specialization

Let's work through a concrete numerical trace of how an MoE layer processes a small batch of tokens. This example makes the abstract architecture tangible and helps you see exactly where the parameter count, capacity limits, and routing decisions come together.

Suppose we have a small MoE layer with the following configuration: $d_{\text{model}} = 4$, $d_{\text{ff}} = 8$, 4 experts, and top-1 routing. We have a batch of 8 tokens, and the routing scores (which we assume have already been computed by the router) assign each token to one expert:

• Tokens 0, 2, 4: assigned to Expert 0 (3 tokens)
• Tokens 1, 5: assigned to Expert 1 (2 tokens)
• Token 3: assigned to Expert 2 (1 token)
• Tokens 6, 7: assigned to Expert 3 (2 tokens)

With a capacity factor of $C = 1.25$ and 4 experts over 8 tokens, the capacity per expert is $K = 1.25 \times (8/4) = 2.5$, which rounds down to 2. Expert 0 is assigned 3 tokens but has capacity for only 2. Token 4, which has the lowest router confidence score among Expert 0's three assigned tokens, is dropped. It will pass through the MoE layer unchanged via the residual connection.

Now let's trace the computation for token 0, which is routed to Expert 0. Token 0's hidden representation is a 4-dimensional vector, let's call it $x_0 = [0.5, -0.3, 0.8, 0.2]$. Expert 0 applies its weight matrices:

First, the up-projection: $h = W_1 x_0 + b_1$, creating an 8-dimensional intermediate vector.

Then, the activation: $a = \sigma(h)$, where $\sigma$ applies SiLU element-wise.

Finally, the down-projection: $y_0 = W_2 a + b_2$, creating the 4-dimensional output.

This output $y_0$ is then scaled by the router's confidence weight for Expert 0 (let's say 0.87) and added to the residual: the final output for token 0 is $x_0 + 0.87 \cdot y_0$.

Token 4, the dropped token, skips the expert computation entirely. Its output is simply $x_4 + 0 = x_4$: the residual connection carries the unchanged representation to the next layer. This means the MoE layer contributes nothing to token 4's representation, which is the cost of token dropping.

The key observation from this trace is that each expert processes its assigned tokens independently and in parallel. Expert 0 processes tokens 0 and 2 simultaneously (as a batch of size 2). Expert 1 processes tokens 1 and 5. Expert 2 processes token 3. Expert 3 processes tokens 6 and 7. All four experts run at the same time on different hardware units. The total wall-clock time for this MoE layer is determined by the slowest expert, which in this case is Expert 0 (processing 2 tokens) rather than Expert 2 (processing 1 token). This illustrates why load balancing matters: uneven load means some hardware is busy while other hardware waits, reducing efficiency.