Feed-Forward Networks › Key Parameters

When implementing or configuring feed-forward networks in transformers, these parameters control capacity and efficiency:

• d_model (input/output dimension): The embedding dimension that the FFN preserves. Typical values range from 256 (small models) to 12288 (GPT-3 scale). This dimension must match the attention layer output and determines the FFN's interface with the rest of the transformer.

• d_ff (hidden dimension): The expanded dimension of the intermediate representation. The standard choice is $d_{ff} = 4 \times d_{\text{model}}$, though modern architectures like LLaMA use ratios around 2.7x when combined with gated linear units. Larger values increase expressiveness but proportionally increase parameters and compute.

• activation: The nonlinear function applied element-wise after the first linear layer. ReLU was used in the original transformer, but GELU has become the standard for encoder models (BERT, RoBERTa) and SiLU/Swish for decoder models (LLaMA, GPT-NeoX). The choice affects gradient flow, sparsity patterns, and training stability.

• use_bias: Whether to include bias terms $b_1$ and $b_2$. Some modern architectures (LLaMA, PaLM) omit biases entirely to reduce parameters and simplify quantization. The impact on model quality is typically minimal.

• dropout (not shown in our implementation): Dropout rate applied to the hidden representation after activation. Values of 0.1-0.2 are common during training to prevent overfitting. Set to 0 during inference.

• initialization: Weight initialization scale affects training stability. Xavier/Glorot initialization (scaling by $\sqrt{2/(d_{in} + d_{out})}$) is standard. Some architectures use scaled initialization for residual paths to maintain signal magnitude through deep networks.