Top-K Routing › Top-2 Routing

The most widely used configuration in modern MoE architectures is top-2 routing, which selects the two experts with the highest routing scores and combines their outputs as a weighted sum. This single change from selecting one expert to selecting two has a surprisingly large impact on model behavior. Mixtral and GShard use top-2 routing, as do several other production systems. This reflects a community-wide finding that the benefits of selecting two experts consistently outweigh the costs.

The philosophical shift from top-1 to top-2 is worth pausing on before we get into the mechanics. Top-1 routing treats expert selection as a categorical decision: a token belongs to the domain of exactly one expert, and the model must commit fully. Top-2 routing treats it as a soft assignment: a token likely belongs primarily to one expert's domain, but acknowledges that a second expert may have something useful to contribute. This mirrors how human concepts work. The word "photosynthesis" is primarily a biological term, but it has chemical and physical dimensions too. Routing it to both a "biology" expert and a "chemistry" expert may produce a richer representation than routing it to either expert alone.

At the same time, top-2 routing doubles the per-token compute cost relative to top-1. This is not free. For a fixed total compute budget, the choice of top-2 over top-1 means either using smaller experts, using fewer experts, or accepting a higher overall cost. The empirical finding that top-2 consistently outperforms top-1 in head-to-head comparisons suggests that the representational benefits outweigh the compute costs, but this is a design decision worth making consciously.

The computation for top-2 routing starts identically to top-1: we compute the same routing logits using the same linear transformation $\mathbf{h} = W_r \mathbf{x}$. The difference lies entirely in what we do next. Instead of selecting just the maximum, we identify both the highest and second-highest scoring experts, calling their indices $k_1$ and $k_2$ respectively.

Before combining the outputs of these two experts, we need to determine how much weight to give each one. Simply using their raw logit values would be problematic: the logits are unconstrained real numbers that don't have a natural interpretation as combination weights. We need the weights to be non-negative and to sum to 1, so that the combined output is a proper convex combination of the two expert outputs. The solution is to apply softmax, but with a necessary design choice: we apply softmax only over the two selected experts, not over all experts.

The gating weights for these experts are computed by applying softmax only over the selected experts:

$$\begin{aligned} g_1 &= \frac{\exp(h_{k_1})}{\exp(h_{k_1}) + \exp(h_{k_2})} \\ g_2 &= \frac{\exp(h_{k_2})}{\exp(h_{k_1}) + \exp(h_{k_2})} \end{aligned}$$

where:

• $g_1, g_2$: the normalized gating weights for the first and second selected experts, both in the range $(0, 1)$
• $h_{k_1}, h_{k_2}$: the raw routing logits for the first and second selected experts
• $\exp(\cdot)$: the exponential function, which ensures all weights are positive and amplifies differences between values
• $\exp(h_{k_1}) + \exp(h_{k_2})$: the sum of exponentials over the selected pair, serving as a normalizing constant that ensures weights sum to 1

Why does applying softmax only over the selected pair matter? If we had applied softmax over all $N$ experts first and then taken the top-2 probabilities, those two probabilities might sum to something like 0.45, not 1.0, because the remaining probability mass would be distributed across the other N-2 experts. The combined output would then be a fraction of the full representation magnitude, not a proper combination. By computing softmax over just the selected pair, we guarantee that $g_1 + g_2 = 1$, making the combination a true weighted average of the two expert outputs.

Why does this formula make sense? Notice that when $h_{k_1} = h_{k_2}$, the two exponentials are equal and the weights work out to exactly $g_1 = g_2 = 0.5$: perfect equal weighting. As $h_{k_1}$ grows larger relative to $h_{k_2}$, $g_1$ grows toward 1 and $g_2$ shrinks toward 0. This is exactly the behavior we want: strongly preferred experts get more weight, weakly preferred experts get less, and the transition is smooth rather than discrete.

The final output combines both expert outputs weighted by these normalized scores:

$$\mathbf{y} = g_1 \cdot E_{k_1}(\mathbf{x}) + g_2 \cdot E_{k_2}(\mathbf{x})$$

where:

• $\mathbf{y}$: the combined output vector for the token, with the same dimension as the input
• $g_1, g_2$: the normalized scalar weights for the two experts
• $E_{k_1}, E_{k_2}$: the functions computed by the selected expert networks applied to the input $\mathbf{x}$
• $\mathbf{x}$: the input token representation

This weighted combination means the final representation is a blend of two expert perspectives. If expert $k_1$ had a much higher routing score than expert $k_2$, then $g_1$ will be close to 1 and $g_2$ will be close to 0, making the output dominated by the first expert. Conversely, if both experts had similar scores, the weights will be close to 0.5 each, giving both experts roughly equal influence. This adaptive weighting allows the model to smoothly interpolate between relying on a single dominant expert and equally blending two perspectives.

Why does top-2 work better than top-1 in many settings? Several factors contribute to the performance difference.

The first is gradient flow. With two active experts per token, gradients reach twice as many expert parameters during each training step. This improves training efficiency and helps experts learn faster, particularly in the early stages of training when routing decisions are noisy. During early training, the router has not yet learned which experts are best for which tokens, so routing decisions are essentially random. With top-2 routing, more experts receive gradient signal during this necessary period, helping the entire expert ensemble learn meaningful specializations more quickly. With top-1 routing, an expert that happens to lose the routing competition for every token in an early batch may receive no gradient signal at all, causing it to fall further behind and creating a self-reinforcing spiral toward expert collapse.

The second is representation flexibility. Tokens often don't fit neatly into single semantic categories. A token representing a technical term in a legal document might benefit from both a "technical/scientific" expert and a "legal language" expert. Top-2 routing allows this blending. Language is inherently ambiguous and contextual, so forcing tokens into exactly one conceptual category can be unnecessarily restrictive. The ability to combine two expert perspectives provides a richer representational palette that better captures the semantic complexity of natural language.

The third is training stability. When only one expert processes each token, small perturbations in the router weights can cause dramatic shifts in which expert learns from which data. A small weight update might flip a routing decision for a large fraction of tokens, causing an abrupt redistribution of training signal that can destabilize learning. With two experts active, there is inherent smoothing: even if the router changes its mind about which expert should be the primary handler of a particular kind of token, that token likely still visits at least one of the same experts it did before. This provides continuity in which experts receive which training signal.

The fourth benefit is that top-2 routing provides better gradient estimates for the router itself. With top-1 routing, the straight-through estimator must approximate the gradient through a complete binary selection. With top-2 routing, the normalized softmax weights $g_1$ and $g_2$ provide a continuous signal: the model can gradually shift weight from the second expert toward the first (or vice versa) by adjusting the router's weights, giving the optimizer a smoother objective surface. This continuity in the gating weights makes the router's parameters easier to optimize.

The computational cost of top-2 routing doubles compared to top-1, since each token now passes through two expert networks instead of one. For a model with 64 experts where each expert's feed-forward network has 1 billion parameters, top-2 routing activates 2 billion parameters per token instead of 1 billion. This is still a dramatic reduction from the 64 billion total expert parameters. Doubling the computation is usually worth the gains in stability and model quality, which is why top-2 has become the de facto standard.