Activation Functions › Swish / SiLU

Swish (also called SiLU, for Sigmoid Linear Unit) was discovered by automated neural architecture search at Google in 2017. It resembles GELU but is simpler in formulation and has become the activation function of choice in several important vision architectures.

Swish is defined as:

$$\text{Swish}(z) = z \cdot \sigma(z) = \frac{z}{1 + e^{-z}}$$

where:

• $z$: the pre-activation value
• $\sigma(z) = 1/(1 + e^{-z})$: the sigmoid function used as a soft gate

Like GELU, Swish scales the input by a smooth gate function. The difference is that Swish uses sigmoid as the gate rather than the normal CDF. For large positive $z$, $\sigma(z) \to 1$ so Swish approaches the identity. For large negative $z$, $\sigma(z) \to 0$ so Swish approaches zero. The function is non-monotone with a minimum around $z \approx -1.28$ where the output is approximately $-0.28$.

The derivative of Swish is:

$$\frac{d\,\text{Swish}(z)}{dz} = \sigma(z) + z \cdot \sigma(z)(1 - \sigma(z)) = \sigma(z)(1 + z(1 - \sigma(z)))$$

where:

• $\sigma(z)$: the sigmoid gate value at $z$
• $z(1 - \sigma(z))$: the product contribution from the input-dependent component

This derivative is smooth everywhere and never hard-zeros the gradient, which aids gradient flow in deep networks.

Swish and GELU are closely related and perform similarly in practice. The main differences are:

• Swish uses sigmoid as the gate function; GELU uses the normal CDF. Both give similar values in the range that matters for neural activations, because sigmoid and the normal CDF have similar shapes.
• Swish has a learnable variant where the sigmoid is scaled by a learnable parameter $\beta$: $z \cdot \sigma(\beta z)$. When $\beta = 1$, this is standard Swish. When $\beta \to 0$, it approaches a linear function. When $\beta \to \infty$, the sigmoid sharpens into a step function and Swish approaches ReLU. This parameterization allows the network to learn the appropriate gate sharpness for each layer.
• GELU is used more in NLP (following BERT's success); Swish/SiLU is more common in vision models and EfficientNet architectures.
• For practical purposes, you can treat GELU and SiLU as interchangeable unless you have specific reasons to prefer one.

PyTorch provides both: torch.nn.GELU() and torch.nn.SiLU().