Dropout › Dropout as Ensemble Approximation

The most illuminating way to understand dropout is through the lens of model ensembles. Ensemble methods train multiple models and average their predictions. They consistently outperform individual models because different models make different mistakes, and averaging predictions cancels out individual errors.

But ensembles are expensive. Training 1000 separate networks requires 1000x the compute and memory. Dropout provides a clever approximation.

Consider a network with $n$ neurons in a layer. Each neuron can be present or absent, giving $2^n$ possible subnetworks (or "thinned networks"). With dropout, every forward pass during training effectively samples one of these $2^n$ architectures. Over the course of training, the parameters of the full network are shared across all these architectures, so training one full network with dropout is approximately equivalent to training an exponential ensemble of smaller networks with shared weights.

At test time, instead of sampling and averaging across all $2^n$ architectures (computationally infeasible), we use a single forward pass through the full network with all neurons active. To correct for the fact that each neuron was only present with probability $(1-p)$ during training, we scale its output down by $(1-p)$. This weight scaling approximates the average prediction of the ensemble.

The ensemble interpretation explains why dropout works so well: it prevents memorization by efficiently approximating a massive ensemble and gaining the benefits of model averaging.

To understand why shared weights across subnetworks is important, consider what happens when you train separate networks. Each network develops its own idiosyncratic solution to the problem, with weights calibrated to work together within that specific network. Averaging the predictions still works, because errors tend to cancel. But each model had to be trained from scratch, requiring separate gradient computations and parameter storage.

With dropout, the shared parameters must simultaneously be good weights across many different subnetworks. A weight that only works well when another specific weight is present will receive inconsistent gradient signals, because sometimes that other weight is dropped. The optimization pressure therefore pushes weights toward values that are individually useful, not just useful in combination. This is the mathematical underpinning of the claim that dropout prevents co-adaptation.