Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

FFN Activation Functions › SiLU/Swish: The Modern Standard

Michael BrenndoerferUpdated June 14, 20257 min read

Language AI HandbookAll chapters

Part 15 of 67Transformer Blocks

Chapter 108FFN Activation Functions › SiLU/Swish: The Modern Standard

Reading section 5 of 15 · SiLU/Swish: The Modern Standard · 7 minSection 5 of 15 · 7 min

Section 5 of 15 · 7 min read

FFN Activation Functions › SiLU/Swish: The Modern StandardOne section · Sections in this chapter5 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

While GELU became the activation of choice for encoder models like BERT, a different function emerged for large decoder models: SiLU, also known as Swish. Google researchers introduced Swish in 2017 through neural architecture search, discovering that this simple formula consistently outperformed ReLU across tasks.

\text{SiLU}(x) = x \cdot \sigma(x) = \frac{x}{1 + e^{-x}}

where:

$x$ : the input value
$\sigma(x)$ : the logistic sigmoid function (defined below)
$e^{-x}$ : the exponential function with negative argument, which rapidly approaches 0 for large positive $x$ and infinity for large negative $x$

The sigmoid function that gates the input is:

\sigma(x) = \frac{1}{1 + e^{-x}}

where:

$\sigma(x)$ : outputs a value in the range $(0, 1)$ , interpretable as a "gate" or probability
For large positive $x$ : $\sigma(x) \to 1$ , so $\text{SiLU}(x) \to x$ (input passes through)
For large negative $x$ : $\sigma(x) \to 0$ , so $\text{SiLU}(x) \to 0$ (input is suppressed)
At $x = 0$ : $\sigma(0) = 0.5$ , so $\text{SiLU}(0) = 0$

The name "SiLU" stands for Sigmoid Linear Unit. This stresses its structure: the input multiplied by its sigmoid. "Swish" was Google's original branding. The two names refer to the same function, though the literature isn't always consistent.

Notice the structural similarity between SiLU and GELU. Both functions multiply the input $x$ by a "gate" value that lies between 0 and 1. In GELU, the gate is $\Phi(x)$ , the Gaussian CDF. In SiLU, the gate is $\sigma(x)$ , the logistic sigmoid. The Gaussian CDF and the logistic sigmoid are very similar functions: both are S-shaped, both map the real line to $(0, 1)$ , and both equal 0.5 at $x = 0$ . The key difference is their transition rate: the logistic sigmoid transitions slightly more quickly, leading to SiLU's slightly deeper negative dip compared to GELU.

SiLU / Swish

An activation function defined as $\text{SiLU}(x) = x \cdot \sigma(x)$ , where $\sigma$ is the sigmoid function. SiLU is smooth, non-monotonic, and unbounded above. It has become the standard activation for decoder-style transformers like LLaMA and Mistral, as well as GPT-NeoX.

In[12]:

Code

def sigmoid(x):
    """Logistic sigmoid function."""
    return 1 / (1 + np.exp(-np.clip(x, -500, 500)))


def silu(x):
    """SiLU/Swish activation."""
    return x * sigmoid(x)


def silu_derivative(x):
    """Derivative of SiLU."""
    sig = sigmoid(x)
    return sig + x * sig * (1 - sig)

SiLU shares GELU's smooth, non-monotonic character. Both have a slight negative dip, both transition smoothly through zero, and both asymptotically approach the identity for large positive values. But the functions differ subtly: SiLU's dip is slightly deeper (minimum around -0.28 compared to GELU's -0.17), and its transition is slightly sharper.

The derivative of SiLU, derived using the product rule, is:

\text{SiLU}'(x) = \sigma(x) + x \cdot \sigma(x) \cdot (1 - \sigma(x))

where:

$\sigma(x)$ : the sigmoid function
$\sigma(x)(1 - \sigma(x))$ : the derivative of the sigmoid, which has a bell-shaped curve centered at $x = 0$
The first term $\sigma(x)$ approaches 1 for large positive $x$ , giving gradient 1
The second term $x \cdot \sigma(x)(1 - \sigma(x))$ adds a positive contribution near $x = 1.5$ , allowing SiLU's derivative to exceed 1

Out[13]:

Visualization

Plot showing SiLU as a smooth curve with negative dip. — SiLU/Swish activation function across [-3, 3]. Like GELU, SiLU is smooth and non-monotonic with a negative dip for moderately negative inputs. The dip is slightly deeper than GELU's, reaching approximately -0.28 at x ≈ -1.28.

Plot showing SiLU derivative as a smooth curve peaking above 1. — SiLU derivative across [-3, 3]. The derivative peaks slightly above 1.0 near x = 1.28, meaning SiLU can amplify gradients for moderate positive inputs, a property not shared by GELU.

Worked Example: Tracing a Value Through SiLULink Copied

Let's trace the same pre-activation values through SiLU to compare with GELU. For $x = 1.0$ :

\sigma(1.0) = \frac{1}{1 + e^{-1}} \approx \frac{1}{1 + 0.368} \approx 0.731

\text{SiLU}(1.0) = 1.0 \times 0.731 \approx 0.731

For $x = -1.0$ :

\sigma(-1.0) = \frac{1}{1 + e^{1}} \approx \frac{1}{1 + 2.718} \approx 0.269

\text{SiLU}(-1.0) = -1.0 \times 0.269 \approx -0.269

Compare these to GELU's values: GELU(1.0) $\approx$ 0.841 and GELU(-1.0) $\approx$ -0.159. SiLU attenuates positive inputs more (0.731 vs. 0.841) and amplifies negative inputs more (-0.269 vs. -0.159). The logistic sigmoid transitions more quickly than the Gaussian CDF, which explains SiLU's larger negative dip and slightly sharper transition region.

In practice, these differences are small enough that switching between GELU and SiLU rarely changes model quality measurably unless you are training at very large scale. The choice between them is largely a matter of convention and ecosystem: GELU for BERT-like models, SiLU for LLaMA-like models.

Why Decoder Models Prefer SiLULink Copied

The preference for SiLU in decoder models like LLaMA and Mistral, as well as Falcon, isn't fully understood theoretically, but several factors contribute:

Simpler computation: SiLU requires only sigmoid and multiplication, avoiding the error function or its approximations. On modern hardware with fast sigmoid implementations, this can be faster than GELU.
Slightly stronger gradients: SiLU's derivative can exceed 1 (peaking around 1.1 near $x = 1.5$ ), which may help gradient flow in very deep networks. GELU's derivative is bounded between 0 and 1.
Empirical performance: In large-scale experiments at the scale of modern LLMs, SiLU consistently matches or slightly outperforms GELU for autoregressive language modeling. The differences are small but measurable.
Pairing with gated linear units: Modern architectures like LLaMA use SiLU specifically within gated linear unit (GLU) variants, where the activation's properties interact with the gating mechanism. We'll explore this in the next chapter.

The historical sequence helps explain this preference. When Meta released LLaMA in early 2023 and described using SiLU in their SwiGLU-based FFN, SiLU's adoption accelerated rapidly across the open-source community. Mistral and Falcon, along with Gemma and Phi, were among the subsequent open models that followed the LLaMA architecture closely, including its SiLU activation. This created a strong network effect: the ecosystem of model weights, fine-tuning code, and inference libraries all assumed SiLU, making it the path of least resistance for new decoder models.

Historical Context: Neural Architecture Search and Swish

SiLU's discovery followed an unusual path. Rather than being derived mathematically or from theoretical principles, Swish was found by automated neural architecture search. Google researchers trained a reinforcement learning agent to propose activation function formulas, evaluating each on a held-out task. The agent discovered that $x \cdot \sigma(\beta x)$ (a parameterized version of SiLU where $\beta$ is learnable or fixed at 1) consistently outperformed ReLU. When $\beta = 1$ , this reduces to the standard SiLU. This discovery highlighted that neural architecture search can find useful building blocks that human intuition might overlook, and contributed to growing interest in automated ML research tools.