Activation Functions › Choosing Activation Functions for Different Architectures

The choice of activation function depends on the architecture, the training setup, and the task. Here are practical guidelines for common scenarios, along with the reasoning behind each recommendation.

Transformer architectures (BERT, GPT, T5, etc.): Use GELU or SiLU. These are the standard choices, supported by extensive empirical evidence across language tasks. The smooth gating property and slight negative region appear to help with the complex feature interactions in self-attention layers. The choice between GELU and SiLU is largely arbitrary in practice: both perform similarly, and GELU has the advantage of historical precedent.

Convolutional networks (ResNets, EfficientNets, YOLO): ReLU is still the safe default for classification. Swish/SiLU is used in EfficientNet with consistent improvements. Mish has shown gains in YOLO object detection. The differences are typically small (1-2% accuracy), but for tasks where you are competing on benchmark leaderboards, using the right modern activation can make a measurable difference.

Recurrent networks (LSTMs, GRUs): Use tanh for the cell state update and sigmoid for the gates. These specific choices are baked into the LSTM and GRU equations and should not be changed without redesigning the gate structure. The bounded range of tanh ($(-1, 1)$) is important for preventing the cell state from growing unboundedly, and the $(0, 1)$ range of sigmoid is required for the gating interpretation where 0 means "forget everything" and 1 means "remember everything."

Fully connected networks without batch normalization: Consider ELU or SELU. ELU's zero-mean property reduces the co-adaptation problem and can improve convergence when you cannot use batch normalization (for example, with very small batch sizes or in sequential models). SELU provides self-normalization if you satisfy its initialization and architecture constraints, which can be particularly valuable when you want a deep network without any normalization layers.

Output layers: The activation function for the output depends entirely on the task:

• Binary classification: Sigmoid, to produce a probability in $(0, 1)$
• Multiclass classification: Softmax, to produce a probability distribution over classes
• Regression: No activation or identity, to allow unbounded outputs
• Bounded regression (e.g., predicting a value between 0 and 1): Sigmoid
• Multi-label classification (each output is an independent binary variable): Sigmoid applied independently to each output

Early stopping and diagnostic: If your network is not training at all (loss not decreasing after the first few batches), the most common culprits are bad learning rate, bad initialization, and dying ReLU. Switching to Leaky ReLU or GELU can quickly rule out the activation as the cause.