FFN Activation Functions › Activation Functions in Practice
Language AI HandbookAll chapters FFN Activation Functions › Activation Functions in PracticeOne section · Sections in this chapter8 / 15 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's simulate how these activations behave in an actual FFN layer, processing typical transformer hidden states:
# Simulate FFN behavior with different activations
np.random.seed(42)
d_model = 768
d_ff = 3072
batch_size = 1000
# Initialize FFN weights (shared across activations)
# Use standard deviation of 1/sqrt(d_model) to produce pre-activations with std ~1
W1 = np.random.randn(d_model, d_ff) / np.sqrt(d_model)
b1 = np.zeros(d_ff)
W2 = np.random.randn(d_ff, d_model) * np.sqrt(2.0 / (d_ff + d_model))
b2 = np.zeros(d_model)
# Generate input batch (simulating post-attention representations)
X = np.random.randn(batch_size, d_model)
# Compute pre-activations
pre_act = X @ W1 + b1
# Apply each activation
activations_dict = {
"ReLU": relu(pre_act),
"GELU": gelu_exact(pre_act),
"SiLU": silu(pre_act),
}
# Compute statistics
def compute_stats(hidden):
"""Compute activation statistics."""
return {
"mean": hidden.mean(),
"std": hidden.std(),
"sparsity": (np.abs(hidden) < 0.01).mean(),
"negative_fraction": (hidden < 0).mean(),
}
stats = {name: compute_stats(h) for name, h in activations_dict.items()}Hidden layer statistics by activation function: Activation Mean Std Sparsity Negative ---------------------------------------------------------- ReLU 0.3991 0.5843 50.4% 0.0% GELU 0.2823 0.5884 2.0% 50.0% SiLU 0.2068 0.5601 1.6% 50.0%
The statistics reveal important differences. ReLU creates the sparsest representations (50% of values near zero) and has no negative outputs. GELU and SiLU have less sparsity and a small fraction of negative outputs. The mean activation is higher for the smooth activations because they don't hard-threshold negative inputs to zero.
These differences in representation density have downstream consequences. The second FFN weight matrix projects the hidden representation back down to . With ReLU, half of the 3072 hidden dimensions are exactly zero, so the matrix multiplication effectively only involves the non-zero dimensions. With GELU or SiLU, most of the 3072 dimensions carry non-zero values, meaning the projection must process a denser representation. This is slightly more expensive but also means the network uses more of its representational capacity on each forward pass.
