Sparse Models › Key Parameters
Michael BrenndoerferUpdated December 31, 20251 min read
Language AI HandbookAll chapters Part 33 of 67Mixture of Experts
Sparse Models › Key ParametersOne section · Sections in this chapter9 / 10 ↓
Sections in this chapter10
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The key parameters for sparse models are:
- n_experts: Number of expert networks in the sparse layer. More experts increase total model capacity but also increase memory requirements and communication overhead.
- top_k: Number of experts to activate per token. Lower values reduce compute cost but may limit the model's ability to combine diverse knowledge.
- d_ff (or expert_dim): Hidden dimension of each expert's feed-forward network. Larger dimensions increase expert capacity but also increase per-expert compute cost.
- load_balance_coefficient: Metric measuring how evenly tokens distribute across experts. Values near 1.0 indicate good balance, while values near 0 suggest severe imbalance requiring intervention.