Weight Initialization › The Symmetry Problem: Why Zero Initialization Fails
Language AI HandbookAll chapters Weight Initialization › The Symmetry Problem: Why Zero Initialization FailsOne section · Sections in this chapter3 / 19 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The most intuitive initialization strategy is to set all weights to the same constant value. Zero seems like a natural, neutral starting point. But this approach is fundamentally broken.
Consider what happens when all weights in a layer are zero. Every neuron receives the same input (zero dot product) and produces the same output. During backpropagation, every neuron also receives identical gradients and applies identical updates. No matter how long you train, every neuron in the layer remains identical to every other. A 512-neuron layer with zero initialization behaves exactly like a 1-neuron layer.
The requirement that neurons in the same layer be initialized differently so they can learn distinct features. Without symmetry breaking, all neurons compute the same function regardless of how wide the network is.
This is the symmetry problem: neurons that start identical stay identical. The network's representational capacity collapses. Any initialization scheme that makes all weights in a layer the same value, whether zero, one, or any constant, triggers this failure mode.
The intuition here is worth dwelling on. Imagine a network designed to recognize different types of objects in images. You want early neurons to detect different low-level features: edges at different orientations, color contrasts, texture patterns. If all neurons start the same and receive the same gradient, they can only ever develop the same feature detector, no matter how many neurons you use or how long you train. The entire point of a wide network is to represent many features in parallel, and this capacity is completely destroyed by symmetric initialization.
The fix is randomness. If we initialize each weight independently from a random distribution, neurons start with different values, receive different gradient signals, and diverge to learn different features. But random initialization introduces a new challenge: what scale should those random values be?