Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Weight Decay › Which Parameters Should Receive Weight Decay?

Michael BrenndoerferJanuary 27, 20264 min read

Language AI HandbookAll chapters

Part 32 of 67Training Optimization

Chapter 235Weight Decay › Which Parameters Should Receive Weight Decay?

Reading section 5 of 11 · Which Parameters Should Receive Weight Decay? · 4 minSection 5 of 11 · 4 min

Section 5 of 11 · 4 min read

Weight Decay › Which Parameters Should Receive Weight Decay?One section · Sections in this chapter5 / 11 ↓

Sections in this chapter11

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

An important practical question is whether to apply weight decay uniformly to all parameters or selectively. In most modern training setups, weight decay is not applied to all parameters equally.

Parameters to Exclude from Weight DecayLink Copied

Certain parameters are typically excluded from weight decay:

Bias terms. Bias parameters shift the activation values of neurons. They do not contribute to the learned feature magnitudes in the same way as weight matrices. Applying weight decay to biases can interfere with the model's ability to represent the correct output scale, and in practice it rarely helps. The bias is a low-dimensional additive offset, and regularizing it toward zero can systematically miscalibrate outputs.

Layer normalization parameters. Layer normalization layers have learned scale ( $\gamma$ ) and shift ( $\beta$ ) parameters. These control the distribution of activations after normalization. Applying weight decay to them can destabilize training by preventing the normalization from correctly calibrating the output scale. In practice, this is one of the more commonly overlooked mistakes that causes subtle training instability.

Embeddings (sometimes). Embedding parameters are weight matrices that map discrete tokens or positions to continuous vectors. Some practitioners exclude them from weight decay because they can be viewed as lookup tables rather than function weights, though this is less universally agreed upon. For large pretraining runs, it is common to include embeddings in weight decay, but for fine-tuning, excluding them or using a very small $\lambda$ for embeddings can prevent degrading carefully trained representations.

Excluding bias and normalization parameters measurably improves training stability and final performance. Most published training configurations for transformers explicitly separate parameters into "weight decay" and "no weight decay" groups before passing them to the optimizer.

The Practical HeuristicLink Copied

A simple and widely used heuristic separates parameters by their dimensionality: parameters with more than one dimension receive weight decay, while one-dimensional parameters do not. Weight matrices are two-dimensional and should receive decay. Bias vectors are one-dimensional and should not. Layer normalization scale and shift vectors are one-dimensional and should not. Embedding matrices are two-dimensional and typically do.

The logic behind this heuristic is that one-dimensional parameters are almost always either offsets (biases) or normalization scalings, which have a different role in the model than the weight matrices that transform feature representations.

In[4]:

Code

def get_parameter_groups(model, weight_decay):
    """Separate parameters into weight decay and no-weight-decay groups."""
    decay_params = []
    no_decay_params = []

    for name, param in model.named_parameters():
        if not param.requires_grad:
            continue
        # Exclude bias terms and normalization parameters
        if param.ndim <= 1 or name.endswith(".bias"):
            no_decay_params.append(param)
        else:
            decay_params.append(param)

    return [
        {"params": decay_params, "weight_decay": weight_decay},
        {"params": no_decay_params, "weight_decay": 0.0},
    ]

Out[5]:

Console

Parameters with weight decay:    2/6
Parameters without weight decay: 4/6

Parameters excluded (bias + LayerNorm):
  0.bias: shape [64]
  1.weight: shape [64]
  1.bias: shape [64]
  2.bias: shape [10]

The output shows exactly which parameters are excluded. Biases and LayerNorm parameters, which have one dimension, are excluded from weight decay. Weight matrices, which are two-dimensional, receive weight decay.

What Happens When You Apply Weight Decay to BiasesLink Copied

To understand why excluding biases matters, consider what weight decay does to a bias term. Suppose a layer has bias $b = 3.0$ that helps shift the activations to the correct scale for the downstream computation. With weight decay, at each step the bias is multiplied by $(1 - \eta \lambda)$ , shrinking it toward zero. Over training, this constant pressure toward zero systematically biases the model's activations in the wrong direction. The task loss gradient will partially compensate, but the result is a tug-of-war where the bias never fully settles at its optimal value. The training dynamic becomes less stable and the final model may have systematically miscalibrated outputs. This effect is subtle in small models but can accumulate significantly in deep networks with many layers.

The same argument applies even more forcefully to layer normalization parameters. The $\gamma$ and $\beta$ parameters of LayerNorm are initialized to values that keep the normalized distribution at unit scale and zero mean. If weight decay shrinks $\gamma$ toward zero, it counteracts the normalization, reducing the effective signal magnitude layer by layer. In deep transformers, this can produce training instability that resembles a learning rate problem but is caused by misapplied regularization.