Dropout › Training vs Inference Mode
Language AI HandbookAll chapters Dropout › Training vs Inference ModeOne section · Sections in this chapter7 / 19 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
One of the most important details about dropout is that it behaves differently during training and inference. During training, dropout stochastically drops neurons. During inference, all neurons are active.
This means your dropout layer must know which mode it's in. In PyTorch, this is controlled via model.train() and model.eval():
import torch
import torch.nn as nn
model = nn.Sequential(
nn.Linear(10, 50), nn.ReLU(), nn.Dropout(0.5), nn.Linear(50, 1)
)
x = torch.ones(1, 10)
torch.manual_seed(0)
# Training mode: dropout is active, outputs vary
model.train()
train_outputs = []
for _ in range(5):
with torch.no_grad():
out = model(x)
train_outputs.append(out.item())
# Eval mode: dropout disabled, output is deterministic
model.eval()
eval_outputs = []
for _ in range(5):
with torch.no_grad():
out = model(x)
eval_outputs.append(out.item())Training mode outputs (stochastic): Pass 1: 0.0920 Pass 2: 0.2330 Pass 3: -0.2499 Pass 4: 0.2512 Pass 5: 0.0203 Eval mode outputs (deterministic): Pass 1: 0.0365 Pass 2: 0.0365 Pass 3: 0.0365 Pass 4: 0.0365 Pass 5: 0.0365
In training mode, each forward pass produces a different output due to the random mask. In eval mode, the output is identical across passes. A common bug in deep learning code is forgetting to call model.eval() before evaluation or inference. This causes the model to apply dropout at test time, introducing random noise into predictions and usually leading to worse and variable performance.
Another subtle point: batch normalization is also affected by model.train() vs model.eval(). During training, batch norm uses mini-batch statistics; during eval, it uses running statistics accumulated during training. As discussed in the batch normalization chapter, forgetting to switch modes is one of the most common bugs in PyTorch code. The model.eval() call correctly affects both dropout and batch normalization layers simultaneously.
A Taxonomy of Mode Errors
To be concrete about what goes wrong when you forget to switch modes, consider the following scenarios.
The first failure mode is using dropout at inference time. The network was trained with inverted dropout, meaning its weights are calibrated for an expected activation level with all neurons present. If you apply dropout at inference, some neurons are zeroed out. The expected activation is now lower than the training configuration, producing predictions with systematically wrong magnitudes. More importantly, the variance across forward passes is high, meaning predictions are inconsistent. If you call this production code repeatedly on the same input, you get different answers each time.
The second failure mode applies to batch normalization. In training mode, each batch's mean and variance are computed from that batch's data. The running statistics stored in the batch norm layer are updated as a moving average. In eval mode, those running statistics are used instead of computing batch statistics. If you evaluate in training mode, small test batches may have very different statistics than the training distribution, leading to poorly normalized activations and unpredictable outputs. If your batch size at inference is 1 (common in real-time systems), the batch statistics are meaningless.
The third failure mode applies to the combination. If a network has both dropout and batch normalization, and you forget to call model.eval(), both misbehave simultaneously in ways that can partially mask each other, making the bug harder to spot. The model may produce results that look roughly plausible but are consistently slightly wrong.
These failure modes are the reason why standard practice is to always call model.eval() at the beginning of any evaluation or inference loop, and to call model.train() at the beginning of any training loop, even if you believe the mode is already correct.