Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Adapter Layers › Adapter Fusion

Michael BrenndoerferDecember 13, 202511 min read

Language AI HandbookAll chapters

Part 35 of 67Parameter-Efficient Fine-tuning

Chapter 264Adapter Layers › Adapter Fusion

Reading section 7 of 10 · Adapter Fusion · 11 minSection 7 of 10 · 11 min

Section 7 of 10 · 11 min read

Adapter Layers › Adapter FusionOne section · Sections in this chapter7 / 10 ↓

Sections in this chapter10

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

When you have multiple tasks and want a model that can handle all of them, you face a basic choice: train a single multi-task model and risk interference between tasks, or train separate adapters for each task and find a principled way to combine them at inference time. Adapter fusion takes the second approach. This provides a learned mechanism for combining multiple task-specific adapters. This technique enables flexible composition of specialized knowledge without the interference problems that often plague joint multi-task training, where improving performance on one task can degrade another.

The key insight behind adapter fusion is that task-specific knowledge, once learned, can be combined through attention rather than through gradient-based joint training. Instead of trying to learn a single set of parameters that works well for all tasks simultaneously (which creates competing objectives during training), we allow each adapter to specialize fully, and then learn how to mix their expertise for new inputs. This separation of the learning problem into two stages, specialization then combination, is what makes adapter fusion work where naive parameter averaging often fails.

The Multi-Adapter ChallengeLink Copied

Imagine you have trained separate adapters for sentiment analysis, named entity recognition, and question answering. Each adapter has learned to transform the base model's representations for its specific task, encoding knowledge about sentiment lexicons, entity boundaries, or answer extraction strategies. Now you encounter a task that requires reasoning about both sentiment and entities, such as analyzing customer reviews to identify which product features receive positive versus negative sentiment.

A simple approach like averaging the adapter outputs treats all adapters as equally relevant for every input, which is clearly wrong. A token like "Apple" in a product review might need strong NER adapter influence to recognize it as an organization entity, while a token like "amazing" should draw primarily from the sentiment adapter to capture its positive polarity. What you need is a learned, input-dependent weighting of adapter contributions: an attention mechanism that can dynamically allocate influence based on what each token requires.

Adapter fusion provides exactly this. Rather than applying a fixed combination strategy, it trains a small attention module that, given the current hidden state, asks each adapter "what do you know that's relevant here?" and weights their contributions accordingly. This learned routing enables the fused model to handle inputs that require specialized knowledge from different task adapters without the adapter outputs interfering with each other.

AdapterFusion ArchitectureLink Copied

AdapterFusion introduces a fusion layer that combines the outputs of multiple pretrained adapters using attention. This approach treats the combination of adapters as itself a learning problem, where the model must discover how to best use each adapter's expertise for a new task.

We want to produce a combined output that is a weighted sum of adapter contributions, where the weights depend on the input context. Given $N$ task-specific adapters with outputs $\{\mathbf{a}_1, \mathbf{a}_2, \ldots, \mathbf{a}_N\}$ for input $\mathbf{h}$ , the fusion layer computes:

\text{AdapterFusion}(\mathbf{h}) = \sum_{i=1}^{N} \alpha_i \cdot \mathbf{v}_i

where:

$\text{AdapterFusion}(\mathbf{h})$ : the combined output for the input, representing a learned mixture of all adapter contributions weighted by their relevance to the current context
$\mathbf{h}$ : the input hidden state, which provides context for determining how to weight the adapters
$\alpha_i$ : the attention weight for the $i$ -th adapter, showing how much that adapter should contribute to this particular input. These weights are non-negative and sum to 1 due to the softmax normalization
$\mathbf{v}_i$ : the value vector for the $i$ -th adapter, projected from the adapter output by the linear map $\mathbf{v}_i = \mathbf{a}_i \cdot \mathbf{W}_V$ . This projection allows the fusion layer to transform adapter outputs into a form suitable for combination
$N$ : the total number of task adapters being fused together

The attention weights $\alpha_i$ are computed via a query-key-value attention mechanism, following the same pattern as the multi-head attention we studied in transformer architectures. The query comes from the hidden state (encoding "what do I need?"), the keys come from adapter outputs (encoding "what do I provide?"), and the dot product between them produces scores that, after softmax normalization, become the weights:

\alpha = \text{softmax}\left(\frac{\mathbf{Q} \cdot \mathbf{K}^T}{\sqrt{d_k}}\right)

where:

$\alpha$ : the vector of attention weights across all $N$ adapters, summing to 1 due to the softmax normalization
$\text{softmax}$ : the function that converts raw scores into a probability distribution, so the weights form a valid convex combination
$\mathbf{Q}$ : the query derived from the hidden state, calculated as $\mathbf{h} \cdot \mathbf{W}_Q$ . The query encodes "what kind of adapter output is needed for this input?"
$\mathbf{K}$ : the keys derived from adapter outputs, calculated as stacking each $\mathbf{a}_i \cdot \mathbf{W}_K$ . Each key encodes "what kind of information does this adapter provide?"
$\sqrt{d_k}$ : the scaling factor based on key dimension $d_k$ to stabilize gradients and prevent the softmax from creating overly peaked distributions that would effectively select only one adapter

The fusion weights $\mathbf{W}_Q$ , $\mathbf{W}_K$ , and $\mathbf{W}_V$ are the only new parameters trained during fusion. All individual adapter weights remain frozen, preserving the task-specific knowledge each adapter has learned. The fusion mechanism discovers how to combine them effectively without any risk of gradient updates corrupting the specialized knowledge that was carefully learned in stage one.

ImplementationLink Copied

Let's implement a basic adapter fusion layer:

In[19]:

Code

class AdapterFusion(nn.Module):
    """
    Combines multiple adapter outputs using learned attention weights.

    Args:
        hidden_dim: Transformer hidden dimension
        num_adapters: Number of adapters to fuse
    """

    def __init__(self, hidden_dim, num_adapters):
        super().__init__()

        self.hidden_dim = hidden_dim
        self.num_adapters = num_adapters

        # Attention parameters for fusion
        self.query_proj = nn.Linear(hidden_dim, hidden_dim)
        self.key_proj = nn.Linear(hidden_dim, hidden_dim)
        self.value_proj = nn.Linear(hidden_dim, hidden_dim)

        self.scale = hidden_dim**-0.5

    def forward(self, hidden_state, adapter_outputs):
        """
        Args:
            hidden_state: Current hidden state [batch, seq_len, hidden_dim]
            adapter_outputs: List of N adapter outputs, each [batch, seq_len, hidden_dim]

        Returns:
            Fused output [batch, seq_len, hidden_dim]
        """
        batch_size, seq_len, _ = hidden_state.shape

        # Stack adapter outputs: [batch, seq_len, num_adapters, hidden_dim]
        stacked = torch.stack(adapter_outputs, dim=2)

        # Query from hidden state: [batch, seq_len, 1, hidden_dim]
        query = self.query_proj(hidden_state).unsqueeze(2)

        # Keys and values from adapter outputs
        # [batch, seq_len, num_adapters, hidden_dim]
        keys = self.key_proj(stacked)
        values = self.value_proj(stacked)

        # Compute attention scores: [batch, seq_len, 1, num_adapters]
        scores = torch.matmul(query, keys.transpose(-2, -1)) * self.scale
        weights = F.softmax(scores, dim=-1)

        # Apply attention: [batch, seq_len, 1, hidden_dim]
        fused = torch.matmul(weights, values)

        # Remove singleton dimension: [batch, seq_len, hidden_dim]
        return fused.squeeze(2), weights.squeeze(2)

Let's test the fusion layer with simulated adapter outputs to verify the shapes and weight normalization:

In[20]:

Code

# Create fusion layer
hidden_dim = 768
num_adapters = 3
fusion = AdapterFusion(hidden_dim, num_adapters)

# Simulate adapter outputs for different tasks
batch_size, seq_len = 2, 16
hidden_state = torch.randn(batch_size, seq_len, hidden_dim)

# Each adapter produces different outputs
adapter_outputs = [
    torch.randn(batch_size, seq_len, hidden_dim) * 0.1,  # Sentiment adapter
    torch.randn(batch_size, seq_len, hidden_dim) * 0.1,  # NER adapter
    torch.randn(batch_size, seq_len, hidden_dim) * 0.1,  # QA adapter
]

# Compute fused output
fused_output, attention_weights = fusion(hidden_state, adapter_outputs)

Out[21]:

Console

AdapterFusion demonstration:

Input hidden state shape: torch.Size([2, 16, 768])
Number of adapters: 3
Fused output shape: torch.Size([2, 16, 768])
Attention weights shape: torch.Size([2, 16, 3])

Sample attention weights (first token):
  Adapter 1 (Sentiment): 0.324
  Adapter 2 (NER):       0.347
  Adapter 3 (QA):        0.329
  Sum: 1.000

The fusion layer produces soft attention weights that sum to 1.0, confirming that the softmax normalization is working correctly. These weights are learned during training on the target task, and different tokens will receive different weight distributions based on their content and context.

Training Adapter FusionLink Copied

Adapter fusion uses a two-stage training process that separates the specialization phase from the combination phase. This separation is basic to why fusion works: if we tried to train all components jointly from the start, the base model and the individual adapters would receive mixed gradients from multiple tasks, leading to the interference problems that the approach is designed to avoid.

Stage one trains individual adapters on their respective tasks with the base model frozen. Each adapter becomes an expert at its task, developing specialized representations that encode task-relevant patterns. Importantly, this stage can be done independently and asynchronously: the sentiment adapter training does not need to know about or coordinate with the NER adapter training. This parallelism is a practical advantage for teams working on multiple tasks.

Stage two trains the fusion layers with both the base model and individual adapters frozen. Only the fusion attention weights ( $\mathbf{W}_Q$ , $\mathbf{W}_K$ , $\mathbf{W}_V$ ) are trainable. This ensures the specialized knowledge in each adapter is preserved: the fusion mechanism cannot corrupt the adapters because they receive no gradient updates during this stage. The fusion layer learns to selectively access the right adapter expertise for each input, token by token.

In[22]:

Code

def prepare_for_fusion_training(model, adapters, fusion_layers):
    """
    Prepare model for AdapterFusion training.
    Freezes base model and adapters, leaves only fusion layers trainable.
    """
    # Freeze base model
    for param in model.parameters():
        param.requires_grad = False

    # Freeze all adapters
    for adapter in adapters:
        for param in adapter.parameters():
            param.requires_grad = False

    # Keep fusion layers trainable
    for fusion in fusion_layers:
        for param in fusion.parameters():
            param.requires_grad = True

    # Count parameters
    fusion_params = sum(
        sum(p.numel() for p in f.parameters()) for f in fusion_layers
    )

    return fusion_params

In[23]:

Code

# Demonstrate parameter counting
dummy_model = nn.Linear(768, 768)  # Simplified base model
adapters = [AdapterModule(768, 64) for _ in range(3)]
fusion_layers = [AdapterFusion(768, 3)]

fusion_param_count = prepare_for_fusion_training(
    dummy_model, adapters, fusion_layers
)

Out[24]:

Console

AdapterFusion training setup:

Fusion layer parameters: 1,771,776
(3 projection matrices x 768 x 768 = 1,769,472)

Trainable parameter status:
  Base model: Frozen
  Adapters: Frozen
  Fusion layers: Trainable

The fusion layer adds about 1.8 million parameters (three $768 \times 768$ projection matrices), which is still far smaller than training the full base model. For three adapters and a 12-layer model with one fusion layer per transformer layer, the total fusion parameters would be $12 \times 1.8\text{M} \approx 21.6\text{M}$ , about 20% of BERT-base. This is higher than a single adapter set but enables flexible combination of three specialized adapters for potentially complex tasks.

Visualizing Adapter FusionLink Copied

Let's visualize how fusion weights might vary across different input tokens in a realistic example:

Out[25]:

Visualization

Heatmap with tokens as columns and three adapters (Sentiment, NER, QA) as rows. Entity tokens show high NER weights while the word amazing shows a high sentiment weight. — Adapter fusion attention weights across tokens in a sample sentence. The model learns to emphasize different task adapters based on token content: entity mentions such as iPhone, Apple, Tim, and Cook receive high NER adapter weights, while opinion words such as amazing activate the sentiment adapter. Non-informative tokens such as punctuation receive more uniform weights across all adapters.

The visualization shows how adapter fusion can learn meaningful, input-dependent attention patterns. Entity tokens like "Apple," "Tim," and "Cook" receive high weights from the NER adapter, which has learned to detect entity boundaries and encode entity type information. The opinion word "amazing" strongly activates the sentiment adapter, which has learned to identify and characterize sentiment-bearing tokens. More neutral or structural tokens like punctuation marks tend to receive more uniform weights. This reflects that they do not strongly favor any particular adapter's expertise. This token-level routing of adapter influence is the key capability that separates adapter fusion from simpler combination strategies.