Dropout › Spatial Dropout for Sequences
Language AI HandbookAll chapters Dropout › Spatial Dropout for SequencesOne section · Sections in this chapter8 / 19 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Standard dropout applies independently to each neuron. For sequence data (text, time series, audio), this approach can be problematic.
Consider processing a sentence through a recurrent or transformer model. Each token position produces a hidden state vector. If we apply standard dropout to these hidden states, we randomly zero different elements at each position. The result is that position 3 might have elements [1, 0, 1, 1, 0, ...] active, while position 7 has [0, 1, 0, 1, 1, ...]. Each position sees a different "version" of the feature space.
This incoherence can be harmful for sequential patterns. A recurrent network trying to track whether "negation" is present needs to see the negation feature consistently across positions, not have it randomly absent at some positions and present at others.
Spatial dropout (also called variational dropout for sequences) addresses this by applying the same mask to all positions in the sequence. For a sequence of hidden states with shape (batch, time, features), we sample a single mask of shape (batch, 1, features) and broadcast it across the time dimension:
where:
- : the masked activation at time step , feature
- : the shared mask for feature (same for all time steps)
- : the original activation at time step , feature
- : dropout probability
With spatial dropout, if feature 5 is dropped at position 1, it's also dropped at positions 2, 3, and so on. The network can no longer "patch" information about a feature by using a different time step that happens to have that feature active. This creates a stronger regularization effect for sequential patterns.
import torch
import torch.nn as nn
class SpatialDropout1D(nn.Module):
"""
Spatial dropout for 1D sequence data.
Drops entire feature channels across all time steps.
"""
def __init__(self, p):
super().__init__()
self.p = p
def forward(self, x):
if not self.training or self.p == 0:
return x
# x shape: (batch, time, features)
batch_size, seq_len, n_features = x.shape
keep_prob = 1.0 - self.p
mask = torch.bernoulli(
torch.full(
(batch_size, 1, n_features),
keep_prob,
device=x.device,
dtype=x.dtype,
)
)
mask = mask.expand(batch_size, seq_len, n_features)
return (x * mask) / keep_prob
torch.manual_seed(42)
x_seq = torch.ones(2, 5, 4) # batch=2, time=5, features=4
standard_drop = nn.Dropout(p=0.5)
spatial_drop = SpatialDropout1D(p=0.5)
standard_drop.train()
spatial_drop.train()
standard_out = standard_drop(x_seq)
spatial_out = spatial_drop(x_seq)Standard dropout output (batch 0): [[2. 2. 2. 2.] [0. 2. 0. 0.] [2. 2. 2. 2.] [0. 0. 2. 0.] [2. 0. 0. 2.]] Spatial dropout output (batch 0): [[2. 2. 0. 2.] [2. 2. 0. 2.] [2. 2. 0. 2.] [2. 2. 0. 2.] [2. 2. 0. 2.]] With spatial dropout, dropped features are consistent across all time steps. With standard dropout, each time step has an independent random mask.
Notice that spatial dropout zeros out entire columns (features) consistently across the time axis, while standard dropout creates a unique pattern at each time step. For sequence models, particularly LSTMs and GRUs where temporal consistency matters, spatial dropout often performs better.
The Variational Dropout Connection
The theoretical motivation for spatial dropout comes from variational inference. Gal and Ghahramani (2016) showed that certain dropout configurations can be interpreted as variational inference in Bayesian neural networks. Specifically, they showed that applying the same dropout mask to all time steps in a recurrent sequence corresponds to a particular approximate posterior distribution over network weights. This is the "variational" in variational dropout.
The intuition is that, for sequential tasks, the same function is applied at every time step. In a recurrent network, the same weight matrix is multiplied by the hidden state at every step. If you want to drop the same feature at every step, you're effectively applying a mask to that weight matrix column, which corresponds to zeroing out a row of for this particular example. This is more meaningful than independently masking individual time steps, because it corresponds to asking "what if this particular weight didn't exist for this example?"
This connection between spatial dropout and Bayesian inference motivates its use beyond just sequential models. Any network where the same transformation is applied multiple times to different parts of the input benefits from consistent masking, because consistency corresponds to a coherent assumption about which weights matter.