Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Gated Linear Units › A Complete Worked Example

Michael BrenndoerferUpdated June 14, 20254 min read

Language AI HandbookAll chapters

Part 15 of 67Transformer Blocks

Chapter 109Gated Linear Units › A Complete Worked Example

Reading section 10 of 16 · A Complete Worked Example · 4 minSection 10 of 16 · 4 min

Section 10 of 16 · 4 min read

Gated Linear Units › A Complete Worked ExampleOne section · Sections in this chapter10 / 16 ↓

Sections in this chapter16

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The formulas we've covered capture the essence of SwiGLU, but seeing actual numbers flow through the computation solidifies understanding. Let's trace through a concrete SwiGLU computation step by step, using deliberately small dimensions so you can follow every multiplication and understand exactly how the gating mechanism transforms an input vector.

Our goal is threefold:

See how the two pathways ( $xW$ and $xV$ ) produce different views of the same input
Observe how Swish's self-gating modulates the first pathway
Watch how the element-wise multiplication combines both pathways into the final representation

In[27]:

Code

# Small example for hand calculation
d_model_tiny = 4
d_ff_tiny = 6

# Initialize with small, traceable values
W_tiny = np.array(
    [
        [0.5, -0.3, 0.2, 0.4, -0.1, 0.3],
        [-0.2, 0.4, 0.1, -0.3, 0.5, -0.2],
        [0.3, -0.1, -0.4, 0.2, 0.3, 0.1],
        [0.1, 0.2, 0.3, -0.1, -0.2, 0.4],
    ]
)

V_tiny = np.array(
    [
        [0.2, 0.4, -0.3, 0.1, 0.3, -0.2],
        [0.4, -0.2, 0.5, -0.1, 0.2, 0.3],
        [-0.1, 0.3, 0.2, 0.4, -0.3, 0.1],
        [0.3, -0.1, 0.1, 0.2, 0.4, -0.4],
    ]
)

W2_tiny = np.array(
    [
        [0.3, -0.2, 0.4, 0.1],
        [-0.1, 0.3, -0.2, 0.4],
        [0.2, 0.1, 0.3, -0.3],
        [0.4, -0.1, 0.2, 0.3],
        [-0.2, 0.4, -0.1, 0.2],
        [0.1, 0.2, 0.4, -0.2],
    ]
)

# Input vector
x_tiny = np.array([1.0, -0.5, 0.8, 0.3])

We have a 4-dimensional input vector that we'll project into a 6-dimensional hidden space. The weight matrices $W$ and $V$ are initialized with small values for readability, and $W_2$ projects back from hidden to input dimension.

Now let's trace through each step of the SwiGLU computation:

Out[28]:

Console

SwiGLU Worked Example
==================================================

Input x: [ 1.  -0.5  0.8  0.3]
Shape: (4,)

Step 1: x @ W = [ 0.87 -0.52 -0.08  0.68 -0.17  0.6 ]

Step 2: Swish(x @ W) = [ 0.6131 -0.1939 -0.0384  0.4513 -0.0778  0.3874]

Step 3: x @ V = [ 0.01  0.71 -0.36  0.53  0.08 -0.39]

Step 4: Swish(x @ W) * (x @ V) = [ 0.0061 -0.1377  0.0138  0.2392 -0.0062 -0.1511]

Step 5: hidden @ W2 = [ 0.1002 -0.0978  0.0222  0.0421]

Final output shape: (4,)
Input and output have the same dimension, ready for residual connection.

The step-by-step output reveals several important patterns:

Step 1 vs Step 2 (Swish effect): Compare $xW$ with $\text{Swish}(xW)$ . Notice how Swish preserves the sign but modulates the magnitude. Larger positive values shrink slightly (due to the sigmoid factor being less than 1 for small values), while negative values get pushed toward zero but can remain slightly negative (unlike ReLU which would zero them completely).

Step 3 (Linear pathway): The values in $xV$ are independent from $xW$ , giving the network a second "view" of the same input through different learned weights.

Step 4 (Multiplicative combination): The final hidden representation emerges from element-wise multiplication. When both pathways have the same sign, the product is positive; opposite signs yield negative products. Near-zero values in either pathway suppress that dimension in the output.

Let's visualize how each step transforms the representation:

Out[29]:

Visualization

Grouped bar chart showing values at each computation step of SwiGLU across hidden dimensions. — Step-by-step visualization of SwiGLU computation. The bars show values at each stage: xW (before Swish), Swish(xW), xV (linear path), and their product (final hidden). The gating mechanism modulates each dimension independently based on both projections.