Gating Networks › Worked Example

Let's trace through routing computation for a concrete example to solidify the concepts. Working through the math shows how formulas produce routing decisions. Suppose we have:

• 4 experts
• Input dimension $d = 4$
• An input token representation: $x = [0.5, -0.3, 0.8, 0.1]$

And a router weight matrix:

$$W_g = \begin{bmatrix} 0.2 & -0.1 & 0.4 & 0.1 \\ 0.3 & 0.2 & -0.2 & 0.5 \\ -0.1 & 0.5 & 0.3 & -0.3 \\ 0.4 & 0.1 & 0.2 & 0.2 \end{bmatrix}$$

Step 1: Compute routing logits

The first step applies the linear projection to compute raw scores for each expert. We multiply the input vector by the weight matrix:

$$G(x) = xW_g$$

For expert 1 (first column), we compute the dot product between the input and the first column of the weight matrix:

$$\begin{aligned} G(x)_1 &= 0.5(0.2) + (-0.3)(0.3) + 0.8(-0.1) + 0.1(0.4) \\ &= 0.1 - 0.09 - 0.08 + 0.04 \\ &= -0.03 \end{aligned}$$

Computing all four expert scores using the same dot product approach:

$$G(x) = [-0.03, 0.30, 0.52, -0.32]$$

Expert 3 has the highest logit (0.52), followed by Expert 2 (0.30). Expert 1 and Expert 4 have negative scores, showing the input doesn't align well with their prototypes.

Step 2: Apply softmax

Next, we convert these raw logits to probabilities. First, compute the exponentials to ensure positive values:

$$\exp(G(x)) = [0.97, 1.35, 1.68, 0.73]$$

Sum: $0.97 + 1.35 + 1.68 + 0.73 = 4.73$

Divide each exponential by the sum to get normalized probabilities:

$$p = [0.21, 0.29, 0.36, 0.15]$$

Notice how the exponential amplified the differences: Expert 3's logit is only 0.22 higher than Expert 2's, but its probability is about 25% higher ((0.36) vs (0.29)).

Step 3: Top-2 selection

The two highest-scoring experts are Expert 3 (0.36) and Expert 2 (0.29). These are the only experts that compute their outputs.

Renormalized weights ensure the selected experts' contributions sum to 1:

$$\tilde{p}_2 = \frac{0.29}{0.36 + 0.29} \approx 0.45$$ $$\tilde{p}_3 = \frac{0.36}{0.36 + 0.29} \approx 0.55$$

The final output is: $y \approx 0.45 \cdot E_2(x) + 0.55 \cdot E_3(x)$

Only experts 2 and 3 are computed, saving roughly 50% of expert computation compared to running all four experts. Expert 3 contributes slightly more to the output due to its higher routing score.