Adapter Layers › Adapter Placement

Where you insert adapters within the transformer block significantly impacts both performance and parameter efficiency. The transformer layer consists of multiple sublayers, each performing a distinct function: the self-attention sublayer captures contextual relationships between tokens, and the feed-forward network sublayer applies position-wise nonlinear transformations. The choice of where to place adapters determines which computations the adapter can influence and how the resulting gradients flow during training. Research has explored several placement strategies that trade expressiveness against efficiency while affecting training stability.

Before diving into specific strategies, it is worth considering what each sublayer contributes to the representation. The attention sublayer reads information from across the entire sequence and writes a context-dependent mixture of values back to each token position. The FFN sublayer applies the same two-layer network independently to each token, serving as a kind of key-value memory that looks up and retrieves learned associations. When we adapt a model for a new task, we might need to adjust either or both of these computations: perhaps the task requires attending to different aspects of the context, or perhaps it requires different feature transformations at each token. The placement strategy determines where in this pipeline the adapter's influence is felt.

The original adapter paper by Houlsby et al. (2019) proposed inserting two adapter modules per transformer layer. This complete approach ensures that adapters can modify representations both after contextual mixing occurs in the attention sublayer and after feature transformation in the feed-forward sublayer. The rationale was conservative: without knowing in advance which sublayer would require adaptation for a given task, inserting adapters after both provides the greatest flexibility. The two insertion points are:

1. After the multi-head attention sublayer: between the attention output and the first layer normalization. This position allows the adapter to modify how attended information is integrated before it enters the residual stream.
2. After the feed-forward network sublayer: between the FFN output and the second layer normalization. This position enables the adapter to adjust the feature transformations applied by the FFN before they are added to the residual.

This placement follows the existing residual connection pattern established in the transformer architecture. To understand how adapters integrate with the standard transformer computation, recall that the original transformer layer computation proceeds as:

$$\begin{aligned} \mathbf{h}_1 &= \text{LayerNorm}(\mathbf{x} + \text{MHA}(\mathbf{x})) \\ \mathbf{h}_2 &= \text{LayerNorm}(\mathbf{h}_1 + \text{FFN}(\mathbf{h}_1)) \end{aligned}$$

where:

• $\mathbf{h}_1$: the intermediate representation after the attention sublayer, capturing contextually mixed information from across the sequence
• $\text{LayerNorm}$: the layer normalization operation that stabilizes training by normalizing activations across the feature dimension
• $\mathbf{x}$: the input to the transformer layer, which may come from the embedding layer or a previous transformer layer
• $\text{MHA}$: the Multi-Head Attention mechanism that computes contextual relationships between all pairs of tokens in the sequence
• $\mathbf{h}_2$: the final output of the transformer layer, ready to pass to the next layer
• $\text{FFN}$: the Feed-Forward Network sublayer that applies position-wise transformations to each token independently

With adapters inserted according to the Houlsby placement strategy, the adapted layer computation becomes:

$$\begin{aligned} \mathbf{h}_1 &= \text{LayerNorm}(\mathbf{x} + \text{Adapter}_1(\text{MHA}(\mathbf{x}))) \\ \mathbf{h}_2 &= \text{LayerNorm}(\mathbf{h}_1 + \text{Adapter}_2(\text{FFN}(\mathbf{h}_1))) \end{aligned}$$

where each component plays the following role:

• $\mathbf{h}_1$: the intermediate representation after the adapted attention sublayer, now incorporating task-specific modifications to the attention output before it joins the residual stream
• $\text{Adapter}_1$: the first adapter module inserted after attention, which learns to modify how attention information should be adjusted for the specific task
• $\text{MHA}(\mathbf{x})$: the Multi-Head Attention output, computed with frozen weights that are never updated
• $\mathbf{h}_2$: the final output of the adapted transformer layer
• $\text{Adapter}_2$: the second adapter module inserted after the feed-forward network, which learns task-specific adjustments to the feature transformations
• $\text{FFN}(\mathbf{h}_1)$: the Feed-Forward Network output, computed with frozen weights

Note that each adapter is applied to the sublayer output before the residual connection with the input is formed. The adapter receives the raw sublayer output, transforms it, and then that transformed output participates in the residual addition. This ordering is deliberate: it allows the adapter to transform the sublayer's contribution while still preserving the original skip connection's ability to pass information through unchanged. The base model's residual structure continues to function exactly as designed; adapters simply intercept and modify the sublayer outputs before they enter that structure.

Later work by Pfeiffer et al. (2021) made a surprising discovery: using a single adapter per layer, placed only after the feed-forward network, achieves comparable performance with half the parameters. This finding challenged the assumption that both insertion points were necessary and led to more efficient adapter configurations becoming standard practice in the field.

The intuition behind this finding connects to what we learned about transformer components in Part XV. Feed-forward networks act as key-value memories that store factual and associative knowledge, while attention layers handle contextual mixing of token information. Task adaptation often requires modifying how the model processes and retrieves information in response to specific inputs, which aligns more closely with the FFN's role as a learned lookup table. When adapting a model for sentiment analysis, for example, the model primarily needs to adjust how features are transformed and emphasized, rather than fundamentally changing which tokens attend to which other tokens.

The attention mechanism is already quite flexible due to the query-key-value decomposition. Different attention heads specialize in capturing different types of relationships, and the linear combination of their outputs provides a rich contextual mixture without the need for additional adaptation. The FFN, in contrast, applies fixed transformation functions that may need task-specific tuning to capture relevant linguistic patterns for the new task. By placing adapters exclusively after the FFN, Pfeiffer et al. directly targeted the component most likely to require modification while eliminating the half of the adapter parameters that produced diminishing returns.

This result has a practical implication that extends beyond parameter counting: training with fewer adapters is faster, uses less memory, and converges more reliably. The reduced complexity of the FFN-only placement makes it the recommended default for most applications, reserving the dual placement for cases where careful ablation shows it is needed.

An alternative to sequential placement is parallel adapters, where the adapter computation runs alongside the original sublayer rather than after it. This design represents a departure from the standard "intercept and modify" approach, instead treating the adapter as an independent contribution to the layer output.

Rather than sequentially transforming the sublayer output, we want the adapter and the original sublayer to operate in parallel on the same input, and then combine their contributions. The formulation for parallel placement following a multi-head attention sublayer is:

$$\mathbf{h}' = \mathbf{x} + \text{MHA}(\mathbf{x}) + s \cdot \text{Adapter}(\mathbf{x})$$

where:

• $\mathbf{h}'$: the output of the layer, combining contributions from the input (via the residual skip), the attention mechanism, and the adapter
• $\mathbf{x}$: the input to the layer, which is the common source for all three pathways. The adapter operates on $\mathbf{x}$ directly, not on the attention output
• $\text{MHA}(\mathbf{x})$: the attention sublayer output, computed from the input as in the standard transformer
• $s$: a scaling factor that controls the adapter's contribution to the final representation. This hyperparameter allows fine-grained control over how much the adapter influences the output, and can be tuned per task
• $\text{Adapter}(\mathbf{x})$: the adapter computation running in parallel with the attention mechanism, operating directly on the layer input rather than on the attention output

Parallel placement can offer computational advantages on hardware that supports concurrent operations, as the adapter and original sublayer can be computed simultaneously. On modern GPUs and TPUs, this parallelism can translate to reduced wall-clock time even though the total computation in floating-point operations is the same. Additionally, parallel placement changes the gradient flow: the adapter receives gradients directly from the layer output without having to backpropagate through the attention computation, which can lead to different optimization dynamics. Some researchers have found that parallel adapters train more stably on certain tasks, though the evidence is mixed.

Parallel adapters can also be viewed as implicitly learning an additive correction to the layer's function. If we write the full layer computation as $\mathbf{h}' = \text{Layer}(\mathbf{x}) + s \cdot \text{Adapter}(\mathbf{x})$, we see that the adapter is learning to produce a vector that, when added to the standard layer output, improves task performance. This is similar in spirit to how LoRA modifies weight matrices: instead of changing the weights, we add a learned correction to the outputs.

The choice of adapter placement involves trade-offs between several competing factors. Understanding these trade-offs helps you make informed decisions for your specific use case:

• Parameter efficiency: Single FFN-only adapters use half the parameters of dual placement. This translates directly to storage savings when maintaining many task-specific adapters, and also reduces memory requirements during training.
• Performance: Dual placement often achieves slightly better results on diverse tasks, particularly tasks that require modifying both contextual mixing and feature transformation. The advantage is typically 0.5 to 1.5 percentage points on standard benchmarks, which may or may not be worth the additional parameter cost depending on your application.
• Training stability: The original dual placement provides more gradient pathways, potentially making optimization easier for difficult tasks with little training data. More adapters mean more places where task-specific gradients can be absorbed, reducing the adjustment each individual adapter needs to make.
• Inference overhead: More adapters mean more sequential computations, increasing latency during inference. For latency-sensitive applications, minimizing the number of adapter forward passes per layer is important.

For most practical applications, the single adapter placement after the FFN provides an excellent balance of efficiency and performance. The savings in parameters and inference time typically outweigh the marginal performance improvements of dual placement, especially when deploying multiple adapters for different tasks.