Dropout › Dropout

Neural networks are powerful function approximators. Given enough parameters, they can memorize training data perfectly, achieving near-zero training loss. But this very capacity creates a fundamental problem: a network that memorizes training examples instead of learning patterns will fail on any new data it hasn't seen before.

This problem, called overfitting, was especially acute in the early deep learning era. Networks had millions of parameters but relatively small datasets. Regularization techniques like L2 weight decay helped, but were often insufficient for very deep or wide networks. In 2014, Srivastava, Hinton, Krizhevsky, Sutskever, and Salakhutdinov introduced dropout, a deceptively simple technique that dramatically improved generalization across many architectures.

The core idea is almost shockingly straightforward: during training, randomly "drop" neurons by setting their outputs to zero. This seemingly destructive intervention forces the network to develop redundant representations, preventing any single neuron from becoming too specialized. The result is a more robust model that generalizes better to new data.

This chapter explores dropout in depth, from its ensemble interpretation to its implementation details, from rate selection to how modern architectures adapt it for attention mechanisms and sequence models. We'll also see how dropout enables a form of uncertainty estimation at inference time that goes well beyond its original regularization purpose.