Adapter Layers › Adapter Layers

While LoRA modifies attention weights through low-rank decomposition and prefix tuning prepends learnable tokens to the input, adapter layers take a different architectural approach: they insert small, trainable neural network modules directly into the transformer's forward pass. These compact bottleneck modules learn task-specific transformations while leaving the original model weights completely frozen. The result is a family of techniques that achieves something remarkable: a single pretrained model can support dozens of distinct tasks simultaneously, with only a tiny fraction of the parameters needing to change per task.

Adapters emerged from a simple observation: if we want to add new capabilities to a pretrained model without disrupting what it already knows, why not add new components rather than modify existing ones? This additive approach separates pretrained knowledge from task-specific adaptations. The pretrained model becomes a static foundation, like a well-tuned musical instrument, and adapters are the small adjustments a performer makes to match a particular venue or style without fundamentally changing the instrument. The base model stays the same; only the thin adaptive layer on top of it changes per task.

Think of adapter layers as the software equivalent of a plugin system. A plugin does not rewrite the host application; it inserts a small module at a specific hook point, and that module intercepts the data flowing through that point and modifies it for a specific purpose. Adapters work exactly this way. They intercept the representations flowing through the transformer's layers, apply learned task-specific transformations, and pass modified representations downstream. The host application, the pretrained model, is untouched.

To understand why this matters in practice, consider the economics of serving language models. A company might need a customer service assistant, a legal document summarizer, a code review tool, and a medical note generator. With full fine-tuning, maintaining these capabilities requires storing and serving four entirely separate copies of a multi-billion parameter model. With adapters, the company stores one frozen base model and four tiny adapter sets. Switching between tasks is as simple as swapping which adapter is active. The storage savings can be three or four orders of magnitude.

In this chapter, we'll explore the adapter architecture in detail, examining how bottleneck modules compress and expand hidden representations, how the residual structure enables near-identity initialization and stable training, and where to place adapters within transformer blocks for maximum effectiveness. We'll also examine how to choose the right bottleneck dimensionality for different tasks, and we'll cover adapter fusion, a technique for combining multiple task-specific adapters to enable flexible multi-task learning. Throughout, we'll see how adapters proved several concepts that shaped the entire parameter-efficient fine-tuning field.