Adapter Layers › Code Implementation
Language AI HandbookAll chapters Adapter Layers › Code ImplementationOne section · Sections in this chapter6 / 10 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's implement adapter layers from scratch and then integrate them into a pretrained transformer model. The goal is to understand the mechanics well enough to adapt any model architecture, not just to call a library function. We will build the adapter module, integrate it into a transformer layer, implement the weight-freezing logic, and verify that our implementation behaves correctly.
import torch.nn as nn # noqa: F401
import torch.nn.functional as F # noqa: F401Building the Adapter Module
We start by implementing a single adapter module as a PyTorch nn.Module. The implementation follows directly from the formula , with the residual connection added in forward. The _init_weights method implements the near-zero initialization strategy:
class AdapterModule(nn.Module):
"""
Bottleneck adapter module with residual connection.
Args:
hidden_dim: The transformer's hidden dimension
bottleneck_dim: The adapter's bottleneck dimension
activation: Activation function ('relu' or 'gelu')
"""
def __init__(self, hidden_dim, bottleneck_dim, activation="relu"):
super().__init__()
# Down-projection: d -> r
self.down_proj = nn.Linear(hidden_dim, bottleneck_dim)
# Up-projection: r -> d
self.up_proj = nn.Linear(bottleneck_dim, hidden_dim)
# Activation function
if activation == "relu":
self.activation = nn.ReLU()
elif activation == "gelu":
self.activation = nn.GELU()
else:
raise ValueError(f"Unknown activation: {activation}")
# Initialize for near-identity at start
self._init_weights()
def _init_weights(self):
# Initialize down-projection with small random values
nn.init.normal_(self.down_proj.weight, std=0.01)
nn.init.zeros_(self.down_proj.bias)
# Initialize up-projection to near-zero for identity initialization
nn.init.normal_(self.up_proj.weight, std=0.01)
nn.init.zeros_(self.up_proj.bias)
def forward(self, x):
# Bottleneck transformation
residual = x
x = self.down_proj(x)
x = self.activation(x)
x = self.up_proj(x)
# Residual connection
return residual + xThe _init_weights method is important for stable training. By initializing both projection matrices with a very small standard deviation (0.01), the initial adapter output is close to zero. Biases are initialized to zero. Combined with the residual connection, this means the model's initial behavior is essentially identical to the frozen pretrained model. Without this initialization, the random adapter weights could produce large, disruptive modifications to the representations from the very first forward pass, making training unstable.
Let's verify our adapter module works correctly and count its parameters:
import torch
# Create an adapter for BERT-base dimensions
hidden_dim = 768
bottleneck_dim = 64
adapter = AdapterModule(hidden_dim, bottleneck_dim)
# Count parameters
total_params = sum(p.numel() for p in adapter.parameters())
trainable_params = sum(
p.numel() for p in adapter.parameters() if p.requires_grad
)
# Test forward pass
batch_size, seq_len = 4, 128
x = torch.randn(batch_size, seq_len, hidden_dim)
output = adapter(x)Adapter configuration: Hidden dimension: 768 Bottleneck dimension: 64 Total parameters: 99,136 Expected parameters: Down projection: 49,216 Up projection: 49,920 Input shape: torch.Size([4, 128, 768]) Output shape: torch.Size([4, 128, 768])
The adapter adds approximately 99,000 parameters, which is only about 0.09% of BERT-base's 110 million parameters. The output shape matches the input shape exactly: the adapter preserves the batch size, sequence length, and hidden dimension throughout its computation. Notice that the total parameter count matches our formula: .
Integrating Adapters into a Transformer Layer
Now let's create a modified transformer layer that includes adapters. This implementation supports the three main placement strategies we discussed, letting easy comparison between Houlsby dual placement, Pfeiffer FFN-only placement, and attention-only placement:
class AdaptedTransformerLayer(nn.Module):
"""
Transformer layer with adapter modules inserted after attention and FFN.
This implements the original Houlsby et al. placement strategy.
"""
def __init__(
self,
hidden_dim,
num_heads,
ffn_dim,
bottleneck_dim,
dropout=0.1,
adapter_placement="both",
):
super().__init__()
# Standard transformer components
self.self_attn = nn.MultiheadAttention(
hidden_dim, num_heads, dropout=dropout, batch_first=True
)
self.ffn = nn.Sequential(
nn.Linear(hidden_dim, ffn_dim),
nn.GELU(),
nn.Linear(ffn_dim, hidden_dim),
nn.Dropout(dropout),
)
self.norm1 = nn.LayerNorm(hidden_dim)
self.norm2 = nn.LayerNorm(hidden_dim)
self.dropout = nn.Dropout(dropout)
# Adapter modules
self.adapter_placement = adapter_placement
if adapter_placement in ["both", "attention"]:
self.adapter_attn = AdapterModule(hidden_dim, bottleneck_dim)
if adapter_placement in ["both", "ffn"]:
self.adapter_ffn = AdapterModule(hidden_dim, bottleneck_dim)
def forward(self, x, attention_mask=None):
# Self-attention sublayer
attn_out, _ = self.self_attn(x, x, x, key_padding_mask=attention_mask)
attn_out = self.dropout(attn_out)
# Apply adapter after attention (if configured)
if self.adapter_placement in ["both", "attention"]:
attn_out = self.adapter_attn(attn_out)
x = self.norm1(x + attn_out)
# Feed-forward sublayer
ffn_out = self.ffn(x)
# Apply adapter after FFN (if configured)
if self.adapter_placement in ["both", "ffn"]:
ffn_out = self.adapter_ffn(ffn_out)
x = self.norm2(x + ffn_out)
return xLet's compare parameter counts for different adapter placements to verify the theoretical analysis:
# Configuration
hidden_dim = 768
num_heads = 12
ffn_dim = 3072
bottleneck_dim = 64
# Create layers with different adapter placements
layer_both = AdaptedTransformerLayer(
hidden_dim, num_heads, ffn_dim, bottleneck_dim, adapter_placement="both"
)
layer_ffn_only = AdaptedTransformerLayer(
hidden_dim, num_heads, ffn_dim, bottleneck_dim, adapter_placement="ffn"
)
layer_no_adapter = AdaptedTransformerLayer(
hidden_dim, num_heads, ffn_dim, bottleneck_dim, adapter_placement="none"
)
# Count parameters
def count_adapter_params(layer):
adapter_params = 0
for name, module in layer.named_modules():
if isinstance(module, AdapterModule):
adapter_params += sum(p.numel() for p in module.parameters())
return adapter_params
def count_total_params(layer):
return sum(p.numel() for p in layer.parameters())Parameter comparison for different adapter placements: Placement Total Params Adapter Params Overhead ----------------------------------------------------------------- Both (Houlsby) 7,286,144 198,272 2.8% FFN only (Pfeiffer) 7,187,008 99,136 1.4% No adapters 7,087,872 0 0.0%
The FFN-only placement uses half the adapter parameters while often achieving comparable performance to the original dual placement. The numbers confirm our theoretical analysis: Houlsby placement adds roughly twice the adapter parameters compared to Pfeiffer, while the base transformer layer parameters remain constant.
Freezing Pretrained Weights
A key aspect of adapter-based fine-tuning is keeping the pretrained weights frozen. The entire point of adapters is that the pretrained model's knowledge is preserved while only the small adapter modules learn task-specific behaviors. This requires explicitly marking all base model parameters as non-trainable and all adapter parameters as trainable. Let's implement a utility to prepare a model for adapter training:
def prepare_model_for_adapter_training(model, adapter_names=["adapter"]):
"""
Freeze all parameters except adapter modules.
Args:
model: The model to prepare
adapter_names: List of substrings to identify adapter parameters
"""
# First, freeze all parameters
for param in model.parameters():
param.requires_grad = False
# Then, unfreeze adapter parameters
unfrozen_count = 0
for name, param in model.named_parameters():
if any(adapter_name in name for adapter_name in adapter_names):
param.requires_grad = True
unfrozen_count += 1
# Count trainable vs total parameters
total_params = sum(p.numel() for p in model.parameters())
trainable_params = sum(
p.numel() for p in model.parameters() if p.requires_grad
)
return {
"total_params": total_params,
"trainable_params": trainable_params,
"trainable_pct": trainable_params / total_params * 100,
"unfrozen_modules": unfrozen_count,
}# Demonstrate on our adapted layer
stats = prepare_model_for_adapter_training(layer_both)Model preparation for adapter training: Total parameters: 7,286,144 Trainable parameters: 198,272 Trainable percentage: 2.72% Parameter status by component: self_attn.in_proj_weight Frozen self_attn.in_proj_bias Frozen self_attn.out_proj.weight Frozen self_attn.out_proj.bias Frozen ffn.0.weight Frozen ffn.0.bias Frozen ffn.2.weight Frozen ffn.2.bias Frozen norm1.weight Frozen norm1.bias Frozen norm2.weight Frozen norm2.bias Frozen adapter_attn.down_proj.weight Trainable adapter_attn.down_proj.bias Trainable adapter_attn.up_proj.weight Trainable adapter_attn.up_proj.bias Trainable adapter_ffn.down_proj.weight Trainable adapter_ffn.down_proj.bias Trainable adapter_ffn.up_proj.weight Trainable adapter_ffn.up_proj.bias Trainable
Only the adapter modules remain trainable, while all original transformer weights are frozen. This is the basic mechanism that enables parameter-efficient fine-tuning: the pretrained model is locked in place, and only the small adapter parameters are exposed to gradient updates from the downstream task.
Visualizing Adapter Behavior
Let's visualize how an adapter transforms hidden representations and how its output magnitude evolves during training:
The training dynamics visualization reveals two key aspects of adapter behavior. First, the adapter starts with negligible influence on the representations: the near-zero initialization ensures the model's initial behavior is essentially identical to the frozen pretrained model. Second, the influence grows gradually and saturates: early training epochs see rapid growth as the adapter discovers the most important task-specific adjustments, and later epochs show diminishing marginal improvement as the adapter refines its learned transformation. This saturation behavior is a healthy sign; it indicates the adapter is not continuing to grow without bound, which would suggest instability.