Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Gated Linear Units › Visualizing GLU Representations

Michael BrenndoerferUpdated June 14, 20255 min read

Language AI HandbookAll chapters

Part 15 of 67Transformer Blocks

Chapter 109Gated Linear Units › Visualizing GLU Representations

Reading section 9 of 16 · Visualizing GLU Representations · 5 minSection 9 of 16 · 5 min

Section 9 of 16 · 5 min read

Gated Linear Units › Visualizing GLU RepresentationsOne section · Sections in this chapter9 / 16 ↓

Sections in this chapter16

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

To understand how GLU transforms representations differently from standard activations, let's visualize the hidden activations for the same inputs processed through different FFN variants:

In[23]:

Code

# Create comparable FFNs
d_model_vis = 64
d_ff_vis = 256

# Standard FFN weights
W1_std = np.random.randn(d_model_vis, d_ff_vis) * np.sqrt(
    2.0 / (d_model_vis + d_ff_vis)
)
W2_std = np.random.randn(d_ff_vis, d_model_vis) * np.sqrt(
    2.0 / (d_ff_vis + d_model_vis)
)

# GLU weights (same initialization for comparison)
W_glu = np.random.randn(d_model_vis, d_ff_vis) * np.sqrt(
    2.0 / (d_model_vis + d_ff_vis)
)
V_glu = np.random.randn(d_model_vis, d_ff_vis) * np.sqrt(
    2.0 / (d_model_vis + d_ff_vis)
)
W2_glu = np.random.randn(d_ff_vis, d_model_vis) * np.sqrt(
    2.0 / (d_ff_vis + d_model_vis)
)


def get_hidden_activations(x, variant="standard"):
    """Get hidden layer activations for visualization."""
    if variant == "standard":
        return relu(x @ W1_std)
    elif variant == "swiglu":
        return swish(x @ W_glu) * (x @ V_glu)
    elif variant == "geglu":
        return gelu(x @ W_glu) * (x @ V_glu)


# Generate a batch of inputs
batch_size = 50
X_batch = np.random.randn(batch_size, d_model_vis)

# Get hidden activations for each variant
hidden_std = np.array([get_hidden_activations(x, "standard") for x in X_batch])
hidden_swi = np.array([get_hidden_activations(x, "swiglu") for x in X_batch])

Out[24]:

Visualization

Heatmap of ReLU FFN hidden activations showing many zero values and binary-like sparsity. — Hidden activations with standard ReLU FFN. Many dimensions are exactly zero (dark vertical bands), creating sparse representations with sharp on/off behavior.

Heatmap of SwiGLU FFN hidden activations showing smoother, more continuous activation patterns. — Hidden activations with SwiGLU FFN. The gating mechanism creates smoother, less sparse patterns with more nuanced activation levels across dimensions.

Let's also compare the distribution of activation values:

Out[25]:

Visualization

Overlaid histograms showing ReLU activations with sharp spike at zero versus SwiGLU with smoother distribution. — Distribution of hidden activation values for standard ReLU FFN versus SwiGLU FFN. ReLU creates a spike at zero (from zeroing negative values) plus a positive tail. SwiGLU produces a smoother, more symmetric distribution with negative values preserved, allowing for richer gradient flow during training.

The key observation is that SwiGLU produces more continuous, less sparse activations. While ReLU creates hard sparsity (many exactly-zero values), SwiGLU allows for small negative values and smoother transitions. This can improve gradient flow during training and allows the network to represent graded feature combinations.

Let's quantify this sparsity difference more precisely by looking at how activation values distribute across magnitude ranges:

Out[26]:

Visualization

Grouped bar chart comparing activation magnitude distributions between ReLU and SwiGLU FFNs. — Sparsity patterns in hidden activations. ReLU creates many exact zeros (high sparsity), while SwiGLU distributes activations more evenly. The near-zero region (-0.1 to 0.1) shows ReLU's hard cutoff versus SwiGLU's smooth transition.