Dropout › Dropout in Transformers
Language AI HandbookAll chapters Dropout › Dropout in TransformersOne section · Sections in this chapter9 / 19 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Modern transformer architectures use dropout in several specific locations, each serving a distinct purpose. Understanding where and why dropout is applied in transformers builds on everything we've covered about the technique.
Transformer models apply dropout in multiple places:
- Attention dropout: Applied to the attention weight matrix after the softmax normalization, preventing the model from relying too heavily on specific token pairs
- Embedding dropout: Applied to input embeddings before they enter the transformer, adding noise at the input level
- Feed-forward dropout: Applied between the two linear layers in each feed-forward block
- Residual dropout: Applied to the output of attention or feed-forward sub-layers before adding the residual connection
Attention Dropout
Self-attention computes attention weights that determine how much each token attends to every other token. These weights form a matrix of shape (seq_len, seq_len) for each attention head. Attention dropout zeros out some of these weights, preventing the model from becoming too reliant on specific token pairs.
The mechanism works after the softmax normalization of attention scores. The attention weight matrix has some entries randomly zeroed and the remaining entries rescaled:
where:
- , : query and key matrices
- : key dimension (for scaling)
- : randomly zeros attention weights before applying to values
This forces the model to spread its attention more broadly. Without attention dropout, a model might learn to always attend to specific tokens for specific patterns. Attention dropout regularizes these attention patterns, forcing more distributed representations.
One interesting property of attention dropout is its interaction with the softmax operation. After softmax, attention weights sum to 1 across all query positions. After dropout, the remaining (non-zeroed) weights are rescaled up by the inverted dropout factor, so they still sum to approximately 1. This means the model still "fully attends" to the remaining tokens, just distributed differently. The effect is that each forward pass forces the model to create a coherent representation using only a subset of available token relationships.
Residual Dropout and the Original Transformer
The original "Attention Is All You Need" paper by Vaswani et al. (2017) applies dropout specifically at two points in each transformer block. The first is after the attention output, before adding the residual connection. The second is after the feed-forward network output, before adding its residual connection. This placement ensures that the gradients flowing through the residual pathway are never affected by dropout, only the learned transformations themselves.
The reason for this choice is subtle. Residual connections, as discussed in the residual connections chapter, allow gradients to flow unchanged through the network. If you applied dropout to the residual connection itself, you would sometimes completely zero out the gradient path through the residual, potentially destabilizing training. Applying dropout to the sub-layer output before the addition preserves gradient flow through the residual while still regularizing the learned transformations.
import torch.nn as nn
class TransformerBlock(nn.Module):
"""Simplified transformer block showing dropout placement."""
def __init__(self, d_model=64, n_heads=4, d_ff=256, dropout=0.1):
super().__init__()
self.attention = nn.MultiheadAttention(
d_model,
n_heads,
dropout=dropout, # attention dropout
batch_first=True,
)
self.feed_forward = nn.Sequential(
nn.Linear(d_model, d_ff),
nn.ReLU(),
nn.Dropout(dropout), # FF intermediate dropout
nn.Linear(d_ff, d_model),
)
self.norm1 = nn.LayerNorm(d_model)
self.norm2 = nn.LayerNorm(d_model)
self.dropout1 = nn.Dropout(dropout) # residual dropout after attention
self.dropout2 = nn.Dropout(dropout) # residual dropout after FF
def forward(self, x):
attn_out, _ = self.attention(x, x, x)
x = self.norm1(x + self.dropout1(attn_out))
ff_out = self.feed_forward(x)
x = self.norm2(x + self.dropout2(ff_out))
return xTransformer block parameters: 49,984 Dropout locations in a transformer block: 1. Attention weights (inside MultiheadAttention, p=0.1) 2. Feed-forward intermediate layer (p=0.1) 3. Residual connection after attention (p=0.1) 4. Residual connection after feed-forward (p=0.1) The original 'Attention Is All You Need' paper used p=0.1 throughout.
The original "Attention Is All You Need" paper applies dropout to the output of each sub-layer before adding the residual connection, plus to the sums of embeddings and positional encodings. Modern variants like BERT and GPT-2 follow similar conventions with p = 0.1.
One important note: in large language models, dropout rates have been reduced or eliminated as model scale has increased. At scales with hundreds of billions of parameters and trillions of training tokens, the implicit regularization from data diversity becomes more important than explicit dropout regularization. Some large models use p = 0.0 throughout. GPT-4 and Claude-class models likely use very little dropout, if any, because the sheer scale of training data provides sufficient regularization.
The Dropout Rate Reduction Trend in Transformers
It is worth understanding why dropout rates have trended down in large models, because this teaches something general about when dropout is and isn't needed.
Recall the fundamental purpose of dropout: preventing co-adaptation by randomly disrupting the connections between neurons, forcing each neuron to learn features that are useful independently. This is most important when the network has much more capacity than the data requires, and when the training data is small enough that the same examples are seen many times during training.
As model scale increases, a curious thing happens. Very large models trained on very large datasets encounter each specific training example only a few times across billions of training steps. When each example appears rarely, the model cannot efficiently memorize it through standard gradient descent, because it would need to specifically remember that example across a vast number of weight updates that push in other directions. The data diversity itself acts as an implicit regularizer.
Large models often benefit from their representations being consistent and strong, rather than noisy and distributed. Dropping neurons in a 175-billion parameter model creates disruptions that may interfere with the formation of coherent, high-level features across the deep stack of layers. The empirical finding is that small dropout rates (or zero dropout) lead to better final model quality at large scale.