Dropout › The Dropout Mask

During a training forward pass, dropout applies a binary mask to the neurons in a layer. Each element of the mask is sampled independently:

$$m_i \sim \text{Bernoulli}(1-p)$$

where:

• $m_i$: the mask value for neuron $i$ (either 0 or 1)
• $p$: the dropout probability (probability of dropping, i.e., setting to zero)
• $1 - p$: the probability that neuron $i$ is kept active (the "keep probability" or "retention rate")

The masked output of the layer is then:

$$\tilde{h}_i = m_i \cdot h_i$$

where:

• $\tilde{h}_i$: the output of neuron $i$ after applying dropout
• $h_i$: the pre-dropout activation of neuron $i$

The mask is resampled independently for every training example in every mini-batch. This means two examples in the same batch see different network architectures. Over the course of training, this continuous resampling trains all $2^n$ subnetworks simultaneously.

An important subtlety is the timing of mask application relative to the activation function. Dropout is typically applied to the output of the activation function, not to the pre-activation values. If applied before the activation, the scaled values would pass through a nonlinearity and produce a different distribution than the intended behavior. Applying dropout to post-activation values zeroes the contribution of a neuron entirely, which cleanly corresponds to removing that neuron from the computation.

The independence of masks across neurons within a layer is also important. If you dropped neurons in correlated groups, some co-adapted patterns might survive even under heavy dropout (because the neurons they depend on tend to be dropped or kept together). Independent sampling ensures that any co-adapted pair is disrupted with probability $p^2 + (1-p)^2$, where the first term is the probability both are dropped and the second is that both survive. For $p = 0.5$, both survive with probability 0.25, meaning 75% of forward passes disrupt any given pair. This is the key pressure against co-adaptation.