Top-K Routing › Top-1 Routing

Top-1 routing is the most extreme form of sparsity available in a Mixture of Experts architecture: each token is sent to exactly one expert, the one with the highest routing score, and all other experts are completely bypassed. This approach is also called hard routing because the decision is binary: a token either goes to an expert or it does not, with no intermediate values allowed. The simplicity of this approach is appealing because it maximizes the efficiency advantage of MoE. When each token visits only one expert, we can scale the number of experts arbitrarily without changing the computational cost for any individual token.

To understand why top-1 routing is so powerful, consider the arithmetic. If you have 64 experts and each expert's feed-forward network contains 1 billion parameters, the model has 64 billion expert parameters in total. But when routing each token to exactly one expert, only 1 billion of those parameters participate in computing the output for any given token. The other 63 billion parameters sit idle for that token, available to be activated for other tokens that the router deems appropriate. This separation between total model capacity (determined by N experts) and per-token computational cost (determined by K activations) is the basic economic advantage of sparse MoE architectures.

Think of top-1 routing as a binary tournament. Each expert submits a "bid" for each token in the form of a routing score. The highest bidder wins the token entirely. The losers receive nothing from that token, no gradient signal, no opportunity to refine their representation of that kind of input. This creates strong selective pressure: to win routing competitions, an expert must learn to be distinctively better than all other experts at some identifiable class of tokens. The result, when training works well, is a collection of specialists with sharply differentiated capabilities.

To understand how top-1 routing works, we will walk through the process step by step. The computation begins by projecting the token's hidden representation through the router's learnable weight matrix to obtain a score for each expert. Specifically, we compute the routing scores (logits) for all experts by projecting the token representation $\mathbf{x}$ through the router's learnable weights $W_r$:

$$\mathbf{h} = W_r \mathbf{x}$$

where:

• $\mathbf{h}$: the vector of routing logits (scores) for all experts, one score per expert
• $W_r$: the router weight matrix of shape $(\text{num\_experts} \times \text{hidden\_dim})$
• $\mathbf{x}$: the input token representation vector of dimension $\text{hidden\_dim}$

This linear transformation is a compatibility function. Each row of the weight matrix $W_r$ can be thought of as a "query vector" for one expert, and the dot product between this query and the token representation measures how well-suited that expert is for processing this particular token. A high dot product means the token's representation is well-aligned with what this expert has learned to handle. Higher scores indicate stronger affinity between the token and the expert.

Why a linear transformation? A more complex routing function, perhaps a small neural network, might produce better routing decisions. The linear router has a practical advantage: it is computationally cheap. Routing must happen before any expert computation, so the overhead of the router should be minimal relative to the cost of the experts themselves. A single matrix multiplication accomplishes this efficiently. Because it operates on the same hidden representation that the model uses for everything else, the linear router can use that representation's full semantic content to make routing decisions.

Then, top-1 routing selects the single expert with the highest score:

$$k^* = \arg\max_i h_i$$

where:

• $k^*$: the index of the selected expert
• $\arg\max_i$: the operation that finds the index $i$ maximizing the value
• $h_i$: the routing logit (score) for the $i$-th expert

The argmax operation identifies the expert with the highest score. This expert then processes the token. The output for that token becomes simply the output of the selected expert:

$$\mathbf{y} = E_{k^*}(\mathbf{x})$$

where:

• $\mathbf{y}$: the output vector for the token
• $E_{k^*}$: the feed-forward network function of the selected expert $k^*$
• $\mathbf{x}$: the input token representation vector

This formula is strikingly clean. The entire computation of the expert layer for this token reduces to a single feed-forward pass through one expert network. There is no mixing, no weighting, no combination: just a direct pass through the winner.

Top-1 routing introduces a significant challenge that is easy to overlook at first: the $\arg\max$ operation is not differentiable. Backpropagation requires gradients, and gradients require the operation to be continuous, but argmax produces a discrete integer index that jumps discontinuously as inputs change. How do gradients flow back through a hard selection?

The standard solution uses the straight-through estimator. It uses different computations for the forward and backward passes, accepting the inconsistency as a pragmatic compromise that enables training. During the forward pass, the token goes through only one expert via true hard $\arg\max$ selection. During the backward pass, the operation is treated as a softmax, allowing gradients to flow back to all experts in proportion to their routing probabilities.

Why does this work? Intuitively, the straight-through estimator provides a smooth surrogate for the discrete selection. Experts with higher routing probabilities receive larger gradient signals, which encourages them to specialize further for the types of tokens that currently prefer them. Experts with lower routing probabilities receive smaller gradients, but they still receive some signal, which prevents them from becoming completely unused. The inconsistency between forward and backward passes is a known theoretical weakness, but empirically it works well enough for MoE training to succeed.

Top-1 routing has another characteristic that can be either a feature or a bug depending on your perspective: it forces complete specialization. Each token must commit fully to one expert's computation, with no blending of perspectives from multiple experts. This creates strong incentives for experts to carve out distinct semantic territories: an expert that specializes narrowly and deeply in a particular kind of input will consistently win the routing competition for that kind of input, while a generalist expert that is merely adequate at everything may rarely be selected.

The flip side of this forced specialization is that it makes the model brittle at boundaries. Consider a token that sits semantically between two categories. With top-1 routing, the model must make a definitive choice about which category's expert to use, potentially losing the fine-grained perspective that a second expert could have contributed. A technical legal term, for instance, might benefit from both a "technical vocabulary" expert and a "legal phrasing" expert, but top-1 routing must choose just one. The model learns to handle these boundary cases as best it can given the hard constraint, but the constraint itself limits expressiveness in a way that top-2 routing avoids.