Expert Networks › Key Parameters
Michael BrenndoerferNovember 13, 20252 min read
Language AI HandbookAll chapters Part 33 of 67Mixture of Experts
Expert Networks › Key ParametersOne section · Sections in this chapter9 / 12 ↓
Sections in this chapter12
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Understanding what each hyperparameter controls helps you reason about MoE configurations you encounter in the literature or need to tune yourself. The key parameters for the Expert implementation are:
- d_model: The dimensionality of input and output tokens. This must match the transformer's hidden size since experts sit inside the transformer block and their outputs flow into residual connections. Changing this value requires changing the entire model, including the expert.
- d_ff: The intermediate expansion dimension within each expert FFN. This controls the representational capacity of each expert. Larger values mean more expressive experts but more memory and compute per activated expert. The standard is 4 times d_model for two-layer FFNs and approximately 2.67 times d_model for SwiGLU variants.
- num_experts: The total number of experts available in the layer. This is the primary lever for trading memory against specialization. More experts means more total parameters and potentially finer-grained specialization, but also more memory and communication overhead.
- capacity_factor: A multiplier determining the maximum number of tokens an expert can process relative to a perfectly balanced load. This is the primary lever for trading routing quality against hardware efficiency. Higher values reduce token dropping but increase memory allocation per expert.