Gradient Clipping › Key Parameters

When implementing gradient clipping in PyTorch, understanding the key parameters helps you configure clipping effectively for your specific training scenario:

• max_norm (for clip_grad_norm_): The maximum allowed L2 norm for the entire gradient vector. Gradients exceeding this threshold are scaled down proportionally. Common values range from 0.5 to 5.0, with 1.0 being a typical default for transformer training.

• clip_value (for clip_grad_value_): The maximum absolute value for any single gradient element. Elements outside $[-\text{clip\_value}, \text{clip\_value}]$ are clamped. Typically set higher than max_norm since it applies element-wise rather than globally.

• norm_type: The type of norm to use for clipping (default: 2 for L2 norm). Can be set to float('inf') for infinity norm or other positive values for different $p$-norms.

• error_if_nonfinite (PyTorch 1.9+): Whether to raise an error if gradients contain NaN or infinity values. Setting to True helps catch gradient explosions early rather than silently propagating corrupted values.

When selecting thresholds, start with an exploratory training run without clipping to observe your gradient norm distribution. Set the threshold at the 90th percentile for conservative clipping, or lower if training remains unstable.