Router Z-Loss › Key Parameters
Michael BrenndoerferUpdated January 4, 20261 min read
Language AI HandbookAll chapters Part 33 of 67Mixture of Experts
Router Z-Loss › Key ParametersOne section · Sections in this chapter9 / 12 ↓
Sections in this chapter12
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The key parameters for the MoE layer with z-loss are:
- aux_loss_coeff: Weighting coefficient for the load balancing loss (typically ~0.01).
- z_loss_coeff: Weighting coefficient for the router z-loss (typically 1e-4 to 1e-2).
- num_experts: Total number of experts in the layer.
- top_k: Number of experts active for each token (usually 1 or 2).