Mixtral › Key Parameters
Michael BrenndoerferUpdated December 31, 20251 min read
Language AI HandbookAll chapters Part 33 of 67Mixture of Experts
Mixtral › Key ParametersOne section · Sections in this chapter14 / 15 ↓
Sections in this chapter15
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The key parameters for Mixtral's MoE implementation are:
- num_experts: Number of expert networks per layer (8 in Mixtral). More experts increase model capacity but also increase memory requirements.
- top_k: Number of experts activated per token (2 in Mixtral). Higher values improve quality but increase compute cost.
- hidden_dim: Model's hidden dimension (4,096). Determines the input/output size of expert networks.
- ffn_dim: Intermediate dimension of each expert FFN (14,336). Larger values give more expert capacity.
- num_layers: Number of transformer blocks with MoE layers (32). Each layer routes tokens to experts independently.