Top-K Routing › Key Parameters

Understanding the key parameters in top-K routing helps you make informed decisions when configuring an MoE layer. These are not simply hyperparameters to tune blindly: each parameter represents a distinct architectural choice that affects capacity and efficiency as well as training dynamics.

The key parameters for the Top-K Routing implementation are:

• num_experts: Total number of experts in the MoE layer. This determines the potential capacity of the MoE layer. More experts means more total parameters and potentially more specialized knowledge, but also more memory requirements and complexity in managing load balance. In practice, values between 8 and 128 are most common. Going beyond 128 experts introduces significant engineering complexity in distributed training without proportional quality gains.
• k: Number of experts to select for each token (typically 1 or 2). This controls the basic trade-off between computational cost and model expressiveness. Lower values of K yield better efficiency and stronger specialization pressure, while higher values provide smoother training and richer token representations. For most applications, K=2 is the right starting point.
• hidden_dim: Dimensionality of the input and output representations. This must match the hidden dimension of the surrounding transformer architecture. The router uses this dimension to compute compatibility scores between tokens and experts. Larger hidden dimensions allow the router to make more fine-grained routing decisions, but the router itself is a single linear layer, so the computational overhead of the routing step is modest.
• intermediate_dim: Dimensionality of the expert feed-forward networks. Larger intermediate dimensions increase the capacity of each individual expert at the cost of higher per-expert computational cost. In transformer architectures, the intermediate dimension is typically 4 times the hidden dimension for dense models, and MoE experts often use the same ratio.