Activation Functions › Mish

Mish, proposed by Diganta Misra in 2019, is a self-regularized non-monotone activation that has been used successfully in computer vision and some NLP applications.

Mish is defined as:

$$\text{Mish}(z) = z \cdot \tanh(\text{softplus}(z)) = z \cdot \tanh\!\left(\ln(1 + e^z)\right)$$

where:

• $z$: the pre-activation value
• $\text{softplus}(z) = \ln(1 + e^z)$: a smooth approximation to the ReLU function that avoids the hard zero
• $\tanh(\cdot)$: the hyperbolic tangent applied to the softplus output

Mish, like Swish and GELU, is non-monotone with a slight negative region near its minimum. Its output is unbounded above (the positive side grows like the identity) but bounded below by approximately $-0.31$. The softplus inner function smoothly transitions from near-zero for large negative inputs to near-linear for large positive inputs, and the tanh applied to it produces a gate that smoothly goes from 0 to 1. The product $z \cdot \text{gate}$ then passes increasingly more of the input as $z$ grows more positive.

The derivative of Mish:

$$\frac{d\,\text{Mish}(z)}{dz} = \tanh\!\left(\text{softplus}(z)\right) + z \cdot \text{sech}^2\!\left(\text{softplus}(z)\right) \cdot \sigma(z)$$

where:

• $\text{sech}^2(\cdot)$: the derivative of tanh
• $\sigma(z)$: the sigmoid function, which is the derivative of softplus

Mish is more computationally expensive than GELU or Swish due to the compound computation involving both a softplus and a tanh. However, benchmarks show it outperforms ReLU on image classification tasks, and it has been adopted in YOLO-based object detection architectures where the small accuracy gain per layer compounds across the many layers in the network. For NLP tasks, the differences compared to GELU are small enough that GELU remains the default.

The key insight behind Mish is that applying tanh to softplus creates a particularly smooth gate: softplus is infinitely differentiable, and applying another smooth function to it produces a gate with very gentle second-order derivatives. This smoothness in the gate function translates to smoothness in the overall activation, which tends to produce better-conditioned optimization landscapes.

The next figure compares GELU, Swish/SiLU, and Mish, showing how these modern smooth activations resemble each other while differing from ReLU in the negative region: