Gating Networks › Key Parameters

Michael BrenndoerferUpdated January 3, 20261 min read
Language AI HandbookAll chapters
Part 33 of 67Mixture of Experts
Chapter 241Gating Networks › Key Parameters
Reading section 8 of 11 · Key Parameters · 1 minSection 8 of 11 · 1 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 241 of 513

Section 8 of 11 · 1 min read

Gating Networks › Key ParametersOne section · Sections in this chapter8 / 11 ↓
Sections in this chapter11
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The key parameters for the Gating Network are:

  • d_model: The dimension of input token representations.
  • num_experts: The total number of experts available in the layer.
  • top_k: The number of experts selected for each token. Common values are 1 or 2.
  • bias: Whether to include a bias term in the linear projection. Typically set to False.