Adapter Layers › Summary

This chapter explored adapter layers as a parameter-efficient approach to fine-tuning pretrained language models. We covered the foundational concepts from module design through multi-task fusion, examining both the theory and the practical implementation details.

Adapter architecture uses a bottleneck design where a down-projection matrix $\mathbf{W}_{down}$ compresses the hidden representation from dimension $d$ to a much smaller bottleneck dimension $r$, a nonlinear activation introduces expressive power, and an up-projection matrix $\mathbf{W}_{up}$ restores the original dimension. The residual connection $\mathbf{h}' = \mathbf{h} + \text{Adapter}(\mathbf{h})$ is necessary: it allows the adapter to learn only the delta needed for the new task while preserving pretrained information in the skip connection, and it enables near-zero initialization for stable training.

Parameter efficiency follows directly from the bottleneck dimension: each adapter contains approximately $2rd$ parameters, which for $r = 64$ and $d = 768$ is roughly 98K parameters compared to over 2M in a single attention layer. A full 12-layer BERT-base with dual adapters adds only about 2.4M trainable parameters, roughly 2.2% of the base model.

Adapter placement can follow the original Houlsby strategy with two adapters per layer (after both attention and FFN sublayers) or the more efficient Pfeiffer approach with a single adapter after the FFN. The Pfeiffer placement often achieves comparable performance with half the parameters, which makes it the recommended default for most applications.

Adapter dimensionality controls the trade-off between capacity and efficiency. Bottleneck dimensions of 32 to 64 provide a good default for most tasks, with the compression ratio forcing adapters to learn generalizable transformations. Very small bottlenecks (8 to 16) provide strong regularization for limited-data settings, while larger bottlenecks (128 to 256) are appropriate for complex tasks requiring substantial deviation from pretrained behavior.

Adapter fusion enables combining multiple task-specific adapters through a learned attention mechanism. The two-stage process, first training individual adapters to specialize, then training small fusion layers to combine them, avoids the interference problems of joint multi-task training. The fusion weights are input-dependent, letting different adapters to contribute different amounts depending on the content of each token.

Inference overhead is the primary practical limitation: unlike LoRA, adapters cannot be merged into base model weights and therefore add latency to every inference call. For latency-sensitive applications, this overhead, typically 5 to 15%, may favor weight-space methods. For applications where modularity and interchangeability are valuable, adapters remain a practical choice.

In the upcoming chapter on PEFT comparison, we will examine how adapters stack up against the other methods we have covered, including LoRA variants, prefix tuning, and prompt tuning, across different tasks and model scales. Understanding the relative strengths of each method will help you make principled choices for your specific use cases.