Dropout › Inverted Dropout
Language AI HandbookAll chapters Dropout › Inverted DropoutOne section · Sections in this chapter5 / 19 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The scaling issue at inference creates a practical problem. If we train with neurons active with probability , we need to remember to scale outputs at test time. Forgetting this step produces incorrect predictions, and it requires different code paths for training and inference.
Inverted dropout solves this by doing the scaling at training time instead. Rather than scaling down at test time, we scale up at training time by dividing the kept activations by :
where:
- : the binary mask (1 = kept, 0 = dropped)
- : the pre-dropout activation
- : the keep probability (we divide by this to compensate for the expected reduction in active neurons)
Now the expected output during training matches the output at inference time. At test time, we simply remove the mask entirely and use all neurons with no scaling adjustment needed. The network produces the same expected values it saw during training.
The scaling follows from the activation probability: if a neuron is active only 50% of the time during training (p = 0.5), its downstream neurons only receive its signal half the time. By scaling the activation up by 2 (dividing by 0.5) whenever it is active, we ensure that downstream neurons see the same expected total input regardless of whether dropout is applied.
This calibration is required for the learned weights to remain valid at inference time. When the network learns its weights during training, it calibrates the magnitude of each weight relative to the typical activation it receives. If a downstream neuron typically receives an input with magnitude 1.0, it will learn a weight calibrated for that scale. With inverted dropout, the expected value of the input stays at 1.0 regardless of dropout rate, so weights are calibrated correctly for inference without any test-time adjustment.
Without inverted dropout, the test-time scaling must exactly match the training configuration. This means the dropout probability must be stored, communicated to inference code, and applied correctly. In production systems, this is an easy place to introduce subtle bugs. Inverted dropout eliminates this operational risk by making training and inference code paths identical except for the mask application.
import numpy as np
def dropout_forward(x, p_drop, training=True):
"""
Apply inverted dropout to input x.
Args:
x: input activations, shape (batch_size, n_features)
p_drop: probability of dropping each unit (0.0 = no dropout)
training: if True, apply dropout; if False, pass through unchanged
Returns:
Tuple of (dropped_output, mask used)
"""
if not training or p_drop == 0.0:
return x, np.ones_like(x)
keep_prob = 1.0 - p_drop
# Sample binary mask: 1 = keep, 0 = drop
mask = (np.random.rand(*x.shape) < keep_prob).astype(float)
# Inverted dropout: scale by 1/keep_prob so test-time needs no adjustment
out = (x * mask) / keep_prob
return out, maskInput: [1. 2. 3. 4. 5. 6. 7. 8.] Avg train output: [0.99 2.01 3.02 3.99 5.01 5.99 6.97 7.99] Test output: [1. 2. 3. 4. 5. 6. 7. 8.] Max absolute difference (train avg vs test): 0.0294
With inverted dropout, the average output over many training passes closely matches the test-time output. This is the key property: the network learns weights calibrated for a specific expected activation level, and both training (on average) and inference produce the same level. Modern deep learning frameworks like PyTorch and TensorFlow implement inverted dropout by default.
