FFN Activation Functions › Key Parameters

When configuring activation functions for transformer FFNs, these parameters and choices affect model behavior:

• activation_type: The activation function to use. Common options are "relu", "gelu", "gelu_approximate", and "silu". Choose based on your model architecture: GELU for encoder models (BERT-style), SiLU for decoder models (LLaMA-style), or ReLU for maximum simplicity and speed.

• approximate (for GELU): Whether to use the tanh approximation instead of the exact error function. The approximation is faster on most hardware and introduces negligible error (< 0.005 maximum). Most production systems use the approximation.

• inplace (framework-specific): Some frameworks allow in-place activation to reduce memory usage. This modifies the input tensor directly rather than creating a new output tensor. Use with caution, as it can cause issues with gradient computation if the original values are needed.

• numerical clipping (for SiLU/sigmoid): Input values should be clipped to prevent overflow in the exponential function. A range of $[-500, 500]$ is typical. Most deep learning frameworks handle this automatically, but custom implementations should include explicit clipping.

• dtype considerations: Activation functions behave differently at different precisions. At float16 or bfloat16, the dynamic range is limited, which can cause issues with the exponential in SiLU or the error function in GELU. Mixed-precision training typically keeps activations in higher precision for stability.