Weight Initialization › Weight Initialization

Training a neural network means finding good values for its weights, but where do you start? The answer matters more than you might expect. If you initialize weights poorly, activations can explode to infinity or vanish to zero before a single gradient update is applied, making learning impossible before it even begins. The right initialization sets the stage for stable, efficient training.

This chapter explores why weight initialization matters and how techniques like Xavier and He initialization solve the problems that plagued early deep networks. We'll derive these methods from first principles, implement them from scratch, see how modern architectures like BERT and GPT approach initialization, and compare empirically how each method performs in practice. By the end, you'll have the conceptual foundations to reason about initialization decisions for any architecture you encounter, and you'll understand precisely when the choice of initialization scheme makes the difference between a network that trains and one that doesn't.