Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Weight Initialization › Bias Initialization

Michael BrenndoerferApril 25, 20252 min read

Language AI HandbookAll chapters

Part 10 of 67Neural Network Foundations

Chapter 70Weight Initialization › Bias Initialization

Reading section 13 of 19 · Bias Initialization · 2 minSection 13 of 19 · 2 min

Section 13 of 19 · 2 min read

Weight Initialization › Bias InitializationOne section · Sections in this chapter13 / 19 ↓

Sections in this chapter19

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

While we've focused on weight initialization, biases also need consideration. The standard practice is to initialize biases to zero.

In[31]:

Code

import torch.nn as nn


def init_biases(layer, method="zeros"):
    """Initialize biases with specified method."""
    if method == "zeros":
        nn.init.zeros_(layer.bias)
    elif method == "small_positive":
        nn.init.constant_(layer.bias, 0.01)
    elif method == "normal":
        nn.init.normal_(layer.bias, mean=0, std=0.01)

Unlike weights, initializing all biases to the same value (zero) does not cause a symmetry problem. The gradient of the loss with respect to a bias depends on the upstream gradient, not on other neurons' biases. Each neuron receives a distinct upstream gradient because its incoming weights are different (due to random weight initialization), so biases update independently from the start.

This is a subtle but important distinction. The symmetry problem for weights arises because two neurons receiving the same input with the same weights produce the same output and therefore receive the same gradient. Biases do not have this issue because the gradient for a bias in layer $\ell$ depends on the product of all upstream activations and weights from layer $\ell$ to the output. Since those upstream weights differ between neurons, the bias gradients differ too.

For ReLU networks, some practitioners use small positive biases (0.01) to ensure neurons start in an active state. This prevents "dead neurons" where all inputs are negative from the start. However, batch normalization and careful learning rate selection have largely eliminated the need for this adjustment in modern architectures.

There are special cases where bias initialization does matter. The output layer bias can be initialized to the log-odds of the class priors (for classification) or the mean target value (for regression), giving the network a sensible starting point that reduces the number of epochs needed to converge. Gate biases in LSTM cells are sometimes initialized to positive values (commonly 1 or 2) to keep the forget gate initially open, allowing gradients to flow more easily through the gate during early training.