Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Gated Linear Units › Parameter Efficiency Analysis

Michael BrenndoerferUpdated June 14, 20254 min read

Language AI HandbookAll chapters

Part 15 of 67Transformer Blocks

Chapter 109Gated Linear Units › Parameter Efficiency Analysis

Reading section 7 of 16 · Parameter Efficiency Analysis · 4 minSection 7 of 16 · 4 min

Section 7 of 16 · 4 min read

Gated Linear Units › Parameter Efficiency AnalysisOne section · Sections in this chapter7 / 16 ↓

Sections in this chapter16

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

One consideration when adopting GLU variants is the parameter cost. The standard FFN has two weight matrices, while GLU-based FFNs have three. Let's analyze this trade-off carefully.

For a standard FFN with hidden dimension $d_{ff}$ , the parameter count (ignoring biases) is:

\text{Params}_{\text{standard}} = 2 \times d_{\text{model}} \times d_{ff}

This comes from two weight matrices:

$W_1 \in \mathbb{R}^{d_{\text{model}} \times d_{ff}}$ : $d_{\text{model}} \times d_{ff}$ parameters
$W_2 \in \mathbb{R}^{d_{ff} \times d_{\text{model}}}$ : $d_{ff} \times d_{\text{model}}$ parameters

For a GLU-based FFN with the same hidden dimension $d_{ff}$ , the parameter count is:

\text{Params}_{\text{GLU}} = 3 \times d_{\text{model}} \times d_{ff}

This comes from three weight matrices:

$W \in \mathbb{R}^{d_{\text{model}} \times d_{ff}}$ : $d_{\text{model}} \times d_{ff}$ parameters (value/Swish projection)
$V \in \mathbb{R}^{d_{\text{model}} \times d_{ff}}$ : $d_{\text{model}} \times d_{ff}$ parameters (gate/linear projection)
$W_2 \in \mathbb{R}^{d_{ff} \times d_{\text{model}}}$ : $d_{ff} \times d_{\text{model}}$ parameters (output projection)

The GLU variant has 50% more parameters for the same hidden dimension. To maintain the same parameter count as a standard FFN, we can solve for the reduced hidden dimension $d'_{ff}$ :

3 \times d_{\text{model}} \times d'_{ff} = 2 \times d_{\text{model}} \times d_{ff}

Solving for $d'_{ff}$ :

d'_{ff} = \frac{2}{3} \times d_{ff}

This means we need to reduce $d_{ff}$ by a factor of $\frac{2}{3}$ to match the parameter count of a standard FFN.

In[17]:

Code

def count_ffn_params(d_model, d_ff, variant="standard"):
    """Count FFN parameters for different variants."""
    if variant == "standard":
        # W1 and W2
        return 2 * d_model * d_ff
    elif variant in ["glu", "swiglu", "geglu"]:
        # W, V, and W2
        return 3 * d_model * d_ff
    else:
        raise ValueError(f"Unknown variant: {variant}")


def equivalent_glu_hidden_dim(d_model, d_ff_standard):
    """Compute d_ff for GLU to match standard FFN parameter count."""
    standard_params = count_ffn_params(d_model, d_ff_standard, "standard")
    # 3 * d_model * d_ff_glu = 2 * d_model * d_ff_standard
    # d_ff_glu = (2/3) * d_ff_standard
    return int((2 / 3) * d_ff_standard)


# Example: GPT-2 small configuration
d_model_gpt = 768
d_ff_standard = 3072  # Standard 4x expansion

d_ff_glu = equivalent_glu_hidden_dim(d_model_gpt, d_ff_standard)
params_standard = count_ffn_params(d_model_gpt, d_ff_standard, "standard")
params_glu_full = count_ffn_params(d_model_gpt, d_ff_standard, "glu")
params_glu_reduced = count_ffn_params(d_model_gpt, d_ff_glu, "glu")

Out[18]:

Console

Parameter count comparison (GPT-2 Small scale):
  d_model: 768
  Standard FFN d_ff: 3072

  Standard FFN params:          4,718,592
  GLU FFN (same d_ff) params:   7,077,888 (+50%)
  GLU FFN (reduced d_ff=2048) params:  4,718,592 (matched)

The comparison shows that keeping the same hidden dimension increases parameters by 50%, while reducing $d_{ff}$ to 2048 brings the parameter count close to the standard FFN. This demonstrates the practical trade-off: you can either accept more parameters for the same hidden capacity, or reduce hidden capacity to match parameters while gaining the expressiveness of gating.

LLaMA and other modern models typically use the reduced hidden dimension approach. For example, LLaMA-7B uses $d_{\text{model}} = 4096$ with $d_{ff} = 11008$ , which is approximately $\frac{8}{3} \times d_{\text{model}}$ rather than the standard $4 \times d_{\text{model}}$ . This gives roughly the same parameter count as a standard FFN with $4 \times$ expansion while gaining the benefits of gated activation.

Out[19]:

Visualization

Bar chart comparing parameter counts of standard FFN, GLU with same hidden dim, and GLU with matched parameters. — Parameter count comparison between standard FFN and GLU variants across different model dimensions. The 'GLU (matched)' variant uses a reduced hidden dimension to match the standard FFN parameter count, while 'GLU (same d_ff)' shows the 50% parameter overhead of using the same hidden dimension.