Residual Connections › Key Parameters

When implementing residual connections in neural networks, several key parameters control their behavior and training dynamics:

• scale ($\alpha$): The residual scaling factor that multiplies $F(x)$ before adding to the skip path. Values less than 1.0 dampen residual contributions, preventing magnitude explosion in very deep networks. Common values range from 0.1 to 1.0, with $1/\sqrt{L}$ (where $L$ is total layers) providing depth-adaptive scaling.

• norm_position: Controls whether layer normalization is applied before the sub-layer (pre-norm) or after the residual addition (post-norm). Pre-norm provides cleaner gradient flow and is preferred for deep networks (24+ layers). Post-norm was the original transformer configuration and may achieve marginally better final performance with careful training.

• rezero: When enabled, initializes the scaling factor to zero, making each block an exact identity function at initialization. The network learns appropriate scaling during training. This eliminates approximation errors at initialization and can speed up early training.

• dim: The hidden dimension of the residual block. Must match between input $x$ and output $F(x)$ for the addition to be valid. When dimensions change, projection shortcuts $W_s x$ are required.

• expansion_factor: In feed-forward networks within transformers, this controls the ratio between inner dimension and model dimension. Typical value is 4, meaning $d_{ff} = 4 \times d_{model}$.

For transformer implementations in PyTorch, these parameters interact with nn.LayerNorm and nn.MultiheadAttention. The choice between pre-norm and post-norm affects the placement of LayerNorm calls, while residual scaling can be implemented as a learnable nn.Parameter or a fixed constant multiplier.