Dropout › Worked Example: Regularizing a Classifier

To make the interaction between dropout and overfitting concrete, let's walk through a worked example. Suppose you're training a text classifier using a simple MLP on top of pretrained word embeddings. Your training set has 500 examples, and your validation set has 200. You're using a two-layer network with 512 hidden units per layer.

Without any regularization, this network has approximately $512 \times d + 512 \times 512 + 512$ parameters, where $d$ is the embedding dimension. If $d$ is 300 (a typical GloVe dimension), that's about 415,000 parameters for 500 examples. This is an 830:1 ratio of parameters to training examples, a strong signal that regularization is needed.

After the first few epochs of training, you observe that training loss drops to near zero while validation loss stagnates at a high level. This is classic overfitting. You add dropout with p = 0.5 after each hidden layer.

When you run again, several things change. Training loss converges more slowly and stabilizes at a higher value (around 0.3 instead of 0.05). Validation loss now decreases throughout training and reaches a lower final value than before. The generalization gap closes from 0.8 to 0.1.

The higher training loss is not a problem. It reflects the fact that the network is now being evaluated in a harder setting (half the neurons are masked each forward pass). The final network is evaluated with all neurons active, and the weights have been calibrated for this configuration by inverted dropout.

You also notice that the optimal number of training epochs changes. Without dropout, early stopping kicks in at epoch 20 because validation loss starts increasing. With dropout, validation loss continues to decrease through epoch 60 before plateauing. This is consistent with the principle that dropout requires more epochs to converge.

If you wanted to push further, you could increase hidden layer size from 512 to 1024 and use the same p = 0.5 dropout. This provides more capacity while maintaining the regularization. The original dropout paper showed that increasing hidden layer size while using dropout often outperforms a smaller network without dropout.