Dropout › Summary

Dropout is more than a regularization trick. It's a training strategy that simultaneously approximates an exponentially large ensemble of neural networks with shared parameters, forces neurons to develop independent feature representations, and enables uncertainty estimation through Monte Carlo sampling at inference time.

The key insights to carry forward:

• Inverted dropout scales activations at training time so inference requires no adjustment, keeping code simple and bug-free
• The ensemble view explains why dropout works: it trains many thinned networks simultaneously, and the full network approximates their averaged prediction
• Spatial dropout applies consistent masks across sequence positions, preserving temporal coherence in sequence models
• Transformers use dropout in multiple locations (attention weights, feed-forward layers, residual connections) at low rates (0.1)
• MC Dropout turns any dropout network into a Bayesian approximation for free, enabling uncertainty estimation
• DropConnect generalizes dropout to individual weights, which provides finer-grained regularization at higher computational cost
• Dropout and batch normalization interact in complex ways; avoid using them in the same computation pathway
• Dropout is complementary to L2 weight decay and data augmentation, addressing different facets of the overfitting problem
• As model scale increases and training data grows, dropout rates are often reduced or eliminated because data diversity provides implicit regularization

In the next chapter, we'll examine gradient clipping, another training stabilization technique that addresses a different failure mode: the exploding gradients that can destabilize training when gradients grow unboundedly during backpropagation.