Architecture Methods › Expert Expansion and Adapter-Based Approaches

Modern large language models have introduced a different flavor of architecture-based continual learning: rather than growing the full network, you grow a small set of lightweight adapters attached to a frozen backbone. This approach, sometimes called parameter-efficient continual learning, dramatically reduces the parameter overhead per task while preserving the core representations in the frozen backbone.

Adapter modules, originally introduced for parameter-efficient fine-tuning, insert small bottleneck layers into a frozen pre-trained network. Each adapter consists of a down-projection to a low-dimensional bottleneck, a non-linearity, and an up-projection back to the original dimension:

$$\text{Adapter}(\mathbf{h}) = \mathbf{h} + W_{\text{up}} \cdot \sigma\!\left( W_{\text{down}} \mathbf{h} \right)$$

where:

• $\mathbf{h}$: the hidden state from the frozen backbone at that layer
• $W_{\text{down}} \in \mathbb{R}^{r \times d}$: the down-projection to bottleneck dimension $r$
• $W_{\text{up}} \in \mathbb{R}^{d \times r}$: the up-projection back to dimension $d$
• $\sigma$: a non-linearity (typically ReLU or GELU)
• The residual connection $+\mathbf{h}$ ensures that the adapter starts as an identity transformation when initialized near zero

In the continual learning setting, each new task gets its own set of adapter modules while the backbone remains frozen. This provides the same zero-forgetting guarantee as progressive networks but with a much smaller per-task footprint. If the adapter bottleneck dimension is $r$ and the backbone has $d$-dimensional hidden states, the adapter adds $2rd$ parameters per layer per task instead of the full $d^2$ parameters of a new column. For large transformers where $d$ might be 4096 and $r$ might be 64, this is a 64-fold reduction in per-task overhead.

Low-Rank Adaptation (LoRA), which we discussed in the Parameter-Efficient Fine-Tuning chapter, decomposes weight updates into low-rank matrices. In the continual learning context, each task trains a set of LoRA adapters while the backbone is frozen. The forward pass for task $t$ at a given weight matrix $W_0$ computes:

$$\mathbf{y} = W_0 \mathbf{x} + \frac{\alpha}{r} B^{(t)} A^{(t)} \mathbf{x}$$

where:

• $W_0 \in \mathbb{R}^{d \times k}$: the frozen pre-trained weight matrix
• $A^{(t)} \in \mathbb{R}^{r \times k}$: the low-rank down-projection for task $t$, initialized randomly
• $B^{(t)} \in \mathbb{R}^{d \times r}$: the low-rank up-projection for task $t$, initialized to zero
• $r$: the rank, controlling expressiveness versus parameter count
• $\alpha$: a scaling hyperparameter

Because $B^{(t)}$ is initialized to zero, the LoRA adapter starts as an identity operation, and the model for task $t$ begins as the frozen pre-trained model. This initialization property makes it easy to deploy the LoRA approach in sequence: each new task starts with the pre-trained model's capabilities and adds just the task-specific delta represented by the low-rank factors.

The key advantage of LoRA-based continual learning is that the stored per-task parameters can be merged with the backbone for inference. If you know the task identity, you add $B^{(t)} A^{(t)}$ to the backbone weights and run inference without any overhead compared to the base model. If you need to switch tasks frequently, you can maintain a library of low-rank deltas and apply them on demand. The storage footprint for the full task library is the backbone (shared) plus $K$ sets of LoRA matrices (one per task), which is far more compact than $K$ full columns.

One limitation of simply training independent LoRA adapters per task is that the adapters can still interfere if they are applied to a shared backbone that evolves. When all prior tasks use the same frozen backbone, there is no interference. But in settings where the backbone itself is allowed to update (for improved performance on recent tasks), the LoRA adapters from prior tasks may no longer function as expected because the backbone they were calibrated against has shifted.

Orthogonal Low-Rank Adaptation (O-LoRA) addresses this by constraining updates to the backbone to lie in subspaces orthogonal to those used by prior task adapters. Specifically, the gradient of the backbone weights is projected to be orthogonal to the span of prior tasks' adapter matrices. This prevents new backbone updates from disrupting the directions that prior adapters rely on, giving a softer version of the hard-freezing guarantee without completely preventing backbone adaptation.