Architecture Methods: Progressive Networks, Expert Expansion

Regularization and replay are two well-studied answers to catastrophic forgetting: the first constrains how much existing weights can move, and the second makes old data available during new training. Both approaches share a fundamental assumption, namely that a single fixed-capacity network will hold all current and future knowledge. Architecture methods reject that assumption. They ask instead: what if the network itself changes as new tasks arrive?

The core insight is that catastrophic forgetting occurs because learning a new task overwrites the parameters used by old ones. If you can quarantine old parameters, or expand the network to accommodate new tasks without disturbing existing representations, forgetting becomes a structural non-issue rather than an optimization challenge. As we discussed in the Regularization Methods chapter, Elastic Weight Consolidation identifies which weights matter most to old tasks and penalizes their movement. Architecture methods take a more radical route by either freezing those weights entirely or routing new tasks through fresh capacity altogether.

Think of the problem in purely structural terms. A single network is like a shared whiteboard. Every time you write something new on it, you risk erasing old content. Regularization gives each piece of content a varying degree of erasure protection. Replay keeps copies of old content nearby so you can restore it if it gets smudged. Architecture methods instead give each piece of content its own dedicated section of the board, or expand the board whenever it fills up. This analogy breaks down in detail, but it captures the philosophical difference: architecture methods solve the forgetting problem spatially rather than temporally.

This chapter covers four families of architecture-based approaches. Progressive networks add lateral connections and new columns for each task, keeping old columns frozen. Expansion methods dynamically grow the network capacity when they detect insufficient resources. Mask-based methods like HAT (Hard Attention to the Task) learn binary attention patterns that protect previously claimed units while working within a fixed total parameter budget. Modular approaches decompose knowledge into reusable components that different tasks can assemble in different combinations. Each family makes different trade-offs between memory growth, forward transfer, and backward transfer.

Progressive networks, introduced by Rusu et al. in 2016 within the context of reinforcement learning transfer, represent the cleanest possible architecture solution to forgetting: every time a new task arrives, you freeze the entire existing network and instantiate a new column. The new column is a fresh network of the same depth as the existing ones. It receives lateral connections from every previous column at every layer, allowing it to draw on previously learned features while leaving old parameters completely untouched.

The motivation behind progressive networks was initially about transfer in reinforcement learning rather than continual learning per se. Rusu and colleagues were training a sequence of Atari games and wanted each new game to benefit from representations learned on prior games, while simultaneously ensuring that training on the new game would not degrade performance on earlier ones. The standard solution at the time was fine-tuning, which provided transfer but destroyed old performance. Progressive networks offered a way to get both benefits, at the cost of growing the network with each new domain.

A progressive network with $K$ tasks consists of $K$ columns. Column $k$ has parameters $\theta^{(k)}$ and produces layer activations $h_i^{(k)}$ at layer $i$. When training on task $k$, all columns $1$ through $k-1$ are frozen. The forward pass through column $k$ at layer $i$ computes:

where:

• $W_i^{(k)}$: the weight matrix for layer $i$ of column $k$, trained on the current task
• $U_i^{(k:j)}$: the lateral connection matrix from layer $i-1$ of column $j$ to layer $i$ of column $k$
• $h_{i-1}^{(j)}$: the activation from layer $i-1$ of a previous column $j$
• $f$: a non-linear activation function (typically ReLU)

The sum over all $j < k$ means that column $k$ can attend to the layer-$(i-1)$ activations of every preceding column. This is a multi-source feature reuse mechanism: the new column sees its own evolving representations and the frozen representations that already encode prior task knowledge.

The lateral connections $U_i^{(k:j)}$ are initialized to zero and trained alongside $W_i^{(k)}$. Initializing them to zero means the new column starts by computing its activations independently, and the lateral pathways open up only as training on task $k$ finds them useful. This is an important design choice: it prevents prior columns from immediately dominating the new column's representation, allowing the new task to establish its own features first before selectively borrowing from what came before.

In the original paper, the lateral connections were implemented using adapter modules rather than raw weight matrices. Specifically, the lateral from column $j$ to column $k$ at layer $i$ passes through a low-rank projection that compresses the hidden-dim-sized activation of column $j$ before injecting it into column $k$. This compression serves two purposes. First, it reduces the number of parameters in the lateral connections, which would otherwise grow quadratically with the number of columns. Second, it allows the projection to learn a kind of relevance filter: the low-rank adapter can selectively amplify the dimensions of the prior column's representation that are useful for the current task while suppressing irrelevant ones.

Freezing prior columns provides an absolute guarantee: no matter what happens during training on task $k$, the outputs of columns $1$ through $k-1$ for any old task input are identical to what they were when those tasks were learned. There is no forgetting in the traditional sense because there is no parameter sharing across time.

This guarantee is fundamentally different from the probabilistic protection offered by regularization. EWC and its relatives use a penalty term to make it costly to move important weights, but the penalty is finite. Sufficiently strong gradients from new tasks can still overwrite old representations, especially when the new task requires moving in a direction that happens to coincide with a direction important for old tasks. Progressive networks eliminate this fragility entirely by making old parameters literally unmodifiable.

The cost is that the number of parameters grows linearly with the number of tasks. After $K$ tasks, you have $K$ columns plus the lateral connection matrices between them. The lateral connections alone total $O(K^2)$ matrices (one for each column-pair at each layer), though in practice they are projected through a lower-rank adapter to keep this manageable. The storage footprint after 100 tasks exceeds 100 times the single-task footprint because of the lateral adapters.

One of the underappreciated strengths of progressive networks is that they implement a structured form of forward transfer. When column $k$ trains, its lateral connections to all prior columns are free to grow from zero to whatever values are helpful. This means the model can actively decide which prior columns' features to borrow at each layer.

Empirically, Rusu et al. observed that the lateral connections tend to be denser (larger weights) at later layers than earlier layers. This makes intuitive sense: early layers in vision networks learn low-level detectors such as edge detectors and color filters that tend to be universal across tasks, so reusing them is natural and expected. Deeper layers encode more task-specific abstractions that may or may not be transferable. When two tasks are semantically related, you expect denser lateral connections in deeper layers, and that is precisely what was observed when training on Atari games with similar visual statistics.

The forward transfer in progressive networks is automatic and unsupervised in the sense that no explicit signal tells the model which tasks are similar. The lateral connections learn the degree of sharing purely from gradient descent on the new task's objective. If borrowing from an old column helps minimize the new task's loss, the lateral connections strengthen. If it does not, they stay near zero.

The linear parameter growth is the defining limitation of progressive networks. If each column has $N$ parameters, after 100 tasks you have 100 columns and roughly $100N$ parameters to store and run inference through. This is acceptable for tens of tasks in constrained domains like robotics or game-playing, but becomes impractical for continual learning across hundreds or thousands of tasks.

A subtler issue is that progressive networks require knowing task identity at inference time. To produce a prediction for an input from task $k$, you run the input through columns $1$ through $k$ and read off column $k$'s output. Without knowing which column to read from, the architecture does not naturally produce a single prediction. This is a general limitation of architecture-based approaches: they often assume task-specific routing at inference time, an assumption that is realistic for clearly delineated task sequences but breaks down in natural open-ended deployment where task boundaries are fuzzy or unknown.

A third limitation is that the architecture assumes all tasks use the same input format and the same output space, or at least a compatible one. If task 1 is image classification and task 5 is depth estimation, you cannot straightforwardly connect them through shared lateral connections. In practice, progressive networks have been most successful in homogeneous settings where all tasks are variations of the same problem type.

Rather than adding a complete new column per task, expansion methods aim to grow the network more surgically. The core idea is to detect when an existing network lacks sufficient capacity for a new task and add capacity exactly where it is needed, leaving intact the parts that are already adequate.

The philosophical motivation here is efficiency. Progressive networks treat every new task as equally demanding of full new capacity, regardless of whether the new task is nearly identical to a prior one (requiring very little new capacity) or completely novel (requiring a full complement of new parameters). Expansion methods try to adaptively allocate capacity based on actual need.

PackNet (Mallya and Lazebnik, 2018) uses iterative pruning and weight freezing to fit multiple tasks into a single network. The lifecycle for each new task proceeds in three phases.

First, the network is trained on the new task to convergence using all available weights. This training uses whatever capacity has not been frozen by prior tasks. Second, a fraction of weights with the smallest absolute value are pruned (zeroed out) to free capacity for future tasks. Pruning by magnitude is a simple but effective heuristic: weights close to zero contribute little to the model's output, so removing them causes minimal performance degradation. Third, the remaining non-zero weights for this task are frozen, protecting them from future modification.

Future tasks can only use the pruned (freed) weights, which are initially random and available for learning. This process repeats for each new task in sequence, and the final network has a fixed total size regardless of the number of tasks.

The appeal of PackNet is that the total parameter count is constant. There is no memory growth as tasks accumulate. The cost is that each successive task has fewer available parameters, since the frozen weights from all prior tasks are unavailable. The capacity allocated to task $k$ is roughly $(1 - \rho)^{k-1}$ times the full network size, where $\rho$ is the pruning fraction per task. For a network with 1 million parameters and a pruning rate of 50%, task 10 has access to only $(0.5)^9 \approx 0.2\%$ of the network, which is severely limited. Choosing a small pruning rate (say 10%) extends the useful life of the network but reserves less capacity per task.

PackNet still requires task identities at inference time to know which subset of weights to use. It applies a binary mask for each task: the forward pass multiplies all weights by the task-specific mask, effectively silencing weights assigned to other tasks. The masks are stored as binary vectors, so their memory overhead is small compared to the weights themselves.

One practical detail is the order of pruning. If you prune aggressively early (to reserve capacity for many future tasks), each task gets abundant room but the early-task models are smaller and potentially less accurate. If you prune conservatively, early tasks are accurate but fewer tasks can fit. This trade-off requires knowing in advance how many tasks the system will encounter, which is not always possible.

A more principled expansion approach is to let the network detect when it needs more capacity and grow accordingly. DEN (Lifelong Learning with Dynamically Expandable Networks, Yoon et al. 2018) uses selective retraining and dynamic expansion together.

When a new task arrives, DEN first identifies which neurons in the network are most relevant to the new task by measuring activation patterns on a small set of new-task samples. It retrains only those relevant neurons, leaving others frozen. If the loss on new-task data remains above a threshold after selective retraining, DEN adds new neurons to accommodate the remaining difficulty. Neurons that are similar (by cosine similarity of weights) to existing neurons are connected to the existing structure rather than initialized randomly, encouraging reuse of learned feature detectors.

The key equations for neuron expansion in DEN involve a reconstruction loss that combines task performance with a regularization term penalizing drift from prior neuron activations. For layer $l$, the loss when expanding is:

where:

• $\mathcal{L}_{\text{task}}^{(t)}$: the supervised loss on the new task $t$
• $\mathcal{S}$: the set of neurons selected for retraining (not newly added ones)
• $h_j^l$: current activation of neuron $j$ at layer $l$
• $\hat{h}_j^l$: reference activation of neuron $j$ before retraining on task $t$
• $\mu$: drift penalty coefficient controlling how much neuron activations are allowed to change

This formulation allows selected neurons to adapt to the new task while discouraging them from drifting far from the representations that made them useful on old tasks. Newly added neurons are unrestricted and can learn new features freely.

The appeal of DEN over simple column expansion is that it is parsimonious. A new task that is very similar to prior tasks may not trigger any expansion at all, relying entirely on existing neurons with selective fine-tuning. Only tasks that cannot be represented by existing capacity cause the network to grow. This means DEN's memory footprint depends on the actual novelty of the task sequence rather than growing unconditionally.

The challenge with DEN is that it has more hyperparameters to set: the threshold for deciding that selective retraining is insufficient, the number of neurons to add when expansion is triggered, and the drift coefficient $\mu$. Getting these right in practice often requires task-specific tuning, and the selective neuron identification step adds computational overhead at the start of each new task.

Hard Attention to the Task (HAT, Serra et al. 2018) is a mask-based architecture method that falls conceptually between expansion and progressive approaches. Rather than growing the network, HAT learns a binary mask over network units for each task and uses these masks to protect important units from modification during future tasks.

The key innovation in HAT is the way it learns these masks end-to-end during task training, rather than deriving them post-hoc (as in PackNet, which prunes after training). This means the mask for each task is shaped by the actual gradient signal on that task's objective, making sure that the protected units are the ones most useful for the task rather than simply the ones with the largest weights.

HAT assigns each task $t$ a learnable embedding vector $\mathbf{e}_t \in \mathbb{R}^L$ where $L$ is the number of layers. For each layer $l$, an attention vector $\mathbf{a}_l^t \in [0, 1]^{d_l}$ is produced by passing the task embedding through a sigmoid:

where:

• $\mathbf{e}_t^l$: the $l$-th component of the task embedding for task $t$
• $s$: a temperature parameter controlling the sharpness of the mask (large $s$ pushes values toward 0 or 1)
• $\sigma$: the sigmoid function

During the forward pass for task $t$, the activation at each layer $l$ is multiplied element-wise by $\mathbf{a}_l^t$, effectively gating which units contribute to the computation for that task. At training time, $s$ starts small (producing soft, continuous masks near 0.5) and is annealed to large values so that at the end of training, each mask entry is approximately binary. At the start, the soft masks allow gradients to flow through all units, making the optimization landscape smooth. At the end, the hard masks provide crisp unit-level protection boundaries.

This annealing strategy prevents gradients from vanishing before the model has learned which units to claim. If you trained with hard binary masks from the start, the gradient would vanish for units where the mask is exactly zero. By starting soft and hardening gradually, HAT maintains trainability throughout while converging to a clean partition.

The mechanism that protects prior tasks is gradient masking. Before updating parameters during training on task $t$, HAT computes a cumulative mask $\mathbf{m}_l^{\text{cum}}$ that is the element-wise maximum of all previous tasks' hard masks:

This cumulative mask identifies units that are important to at least one prior task. Gradient updates to these units are then scaled down by a factor proportional to $1 - \mathbf{m}_l^{\text{cum}}$, effectively freezing units that have been claimed by prior tasks while allowing unclaimed units to be freely modified.

The result is that each task carves out its own subset of network capacity, and the union of all prior task masks acts as an expanding write-protection region. New tasks must work within the remaining unprotected capacity.

This gradient manipulation happens outside the standard backpropagation graph. After computing gradients through standard backprop, HAT applies an element-wise scale to the gradient tensor before the optimizer step. This is a direct surgical intervention: you are not changing the loss function but instead controlling which gradient directions the optimizer is allowed to follow. The effect is mathematically equivalent to projecting the gradient onto the subspace orthogonal to the protected units.

HAT has a key advantage over progressive networks: the network size is fixed. The overhead per task is a single mask vector per layer, which is $O(d_l)$ per layer per task rather than $O(N)$ per task. This is much more memory-efficient. The limitation is that capacity for new tasks shrinks as more masks accumulate. HAT includes a capacity penalty in the training objective to encourage sparse masks that leave room for future tasks:

where:

• $\lambda$: sparsity penalty coefficient
• $\odot$: element-wise multiplication
• The $L_1$ norm encourages the new task's attention to be sparse within the unclaimed region, preserving capacity for tasks that have not yet arrived

The interaction between the two terms in this loss is subtle. The task loss wants to activate many units to achieve good performance. The sparsity penalty wants to activate as few unclaimed units as possible. These are in tension, and the parameter $\lambda$ controls the balance. A large $\lambda$ forces extreme sparsity, potentially degrading per-task accuracy but preserving more capacity for future tasks. A small $\lambda$ allows generous unit allocation per task, improving current accuracy but potentially exhausting capacity sooner.

HAT was evaluated on a sequence of up to eight different classification datasets in the original paper, including permuted MNIST variants, CIFAR-10, and others. The results demonstrated near-zero backward transfer (forgetting) while maintaining competitive forward accuracy. The masks tended to use roughly 20-40% of units per task depending on task complexity, which means a network can realistically support around five to eight tasks before the unclaimed region becomes too small to reliably accommodate a new one.

One practical extension of HAT is to grow the network when the unclaimed fraction falls below a threshold. This hybrid approach combines the mask-based protection of HAT with the growth capability of expansion methods: you use HAT as long as there is sufficient unclaimed capacity, and you expand the backbone when the unclaimed region runs low. This is more practical than pure expansion (which would grow indefinitely) and more scalable than pure HAT (which would degrade on task ten-plus).

Modular architectures take the most compositional view of architecture methods. Instead of a single monolithic network, the system maintains a library of reusable modules, and each task is represented as a particular combination or composition of those modules. The intuition is that tasks often share sub-skills: translation and summarization both require understanding sentence structure; sentiment analysis and topic classification both require word-level representations. If these sub-skills live in separate modules, new tasks that share sub-skills with old ones can reuse those modules without modification.

The modular hypothesis is intellectually appealing because it mirrors how human cognition is described in cognitive science. People are generally understood to have domain-specific mental modules that handle particular types of input (faces, language, spatial reasoning) and that can be combined flexibly for novel problem-solving. Whether artificial neural networks trained from data naturally develop anything like this structure is an open empirical question, but modular architectures try to impose it structurally rather than waiting for it to emerge.

A natural modular architecture is a mixture of experts (MoE), where each task selects a weighted combination of expert networks for its forward pass. As we covered in the Mixture of Experts chapter, each expert is a small network (often a feed-forward layer or a small block), and a router assigns input-dependent weights to determine how much each expert contributes to the output. In the continual learning setting, existing experts are frozen when new tasks arrive and new experts are added as needed. The router for each new task is trained to select appropriate combinations of old experts (for knowledge transfer) and new experts (for task-specific adaptation). The loss for task $t$ with an MoE backbone is:

where:

• $K$: total number of experts (frozen old plus new)
• $g_k^{(t)}(\mathbf{x})$: the router weight assigned to expert $k$ for task $t$ on input $\mathbf{x}$
• $f_k(\mathbf{x})$: the output of expert $k$ on input $\mathbf{x}$
• The weights $g_k^{(t)}$ satisfy $\sum_k g_k^{(t)} = 1$ and are non-negative

The frozen experts from earlier tasks contribute to the new task's predictions in proportion to how useful the router finds them, enabling automatic knowledge transfer without any explicit specification of which old skills are relevant.

One important design question is whether the experts should be frozen completely or allowed to specialize through lightweight adaptation. A common variant uses a small set of fully trainable base experts shared across all tasks and a set of task-specific delta experts that learn residual adjustments. This mirrors the LoRA-style approach to parameter-efficient fine-tuning, where a frozen base model is augmented by low-rank adapters. The advantage is that the base experts can improve as more tasks expose them to diverse data, while the delta experts ensure each task can still express task-specific patterns.

PathNet (Fernando et al., 2017) is an early modular architecture that explicitly represents the network as a directed graph of reusable modules. The network is a grid of $L$ layers, each with $M$ modules. At each layer, the active path uses at most $N \leq M$ modules. A genetic algorithm evolves a population of candidate path configurations, and each candidate path is trained for a few gradient steps. The best-performing path is selected, its modules are frozen, and the process repeats for the next task.

The module output at layer $l$ along path $\mathcal{P}$ is:

where $\mathcal{P}_l \subseteq \{1, \ldots, M\}$ is the set of modules active at layer $l$ along path $\mathcal{P}$.

PathNet enables precise freezing: once a path is selected for a task, those specific modules (not the entire network) are frozen. Modules not on the winning path for any task remain plastic and can be recruited by future tasks. This allows parameter sharing when two tasks select overlapping paths, unlike progressive networks where each task has exclusive columns.

The evolutionary search is computationally expensive, which is PathNet's main practical limitation. For large networks or many tasks, the search itself becomes a bottleneck. Running a genetic algorithm across hundreds of candidate paths, each requiring several gradient steps to evaluate, imposes significant computational overhead that is hard to justify in applications where training speed matters. Later work replaced the evolutionary search with differentiable routing learned through gradient descent, which is both faster and more compatible with standard deep learning infrastructure.

An appealing property of PathNet is that you can visualize which modules were selected for which tasks, producing an interpretable map of knowledge sharing. If tasks A and B share many modules, the learned paths tell you directly that the system found structural overlap between them. This transparency is harder to achieve with mask-based methods where the shared knowledge is encoded in the same weight values rather than in distinct module assignments.

A more principled approach to module composition is meta-learning the routing mechanism itself. Rather than treating the composition as a static assignment (as in PathNet) or a fixed-capacity mixture (as in MoE), meta-learning methods learn how to assemble modules based on task descriptions or small amounts of task data.

One formulation uses an amortized inference network that takes a set of labeled examples from the new task as input and outputs module selection probabilities. This is trained across a distribution of tasks such that the inferred composition generalizes to held-out tasks. The module library is updated between meta-training episodes to ensure it contains modules worth composing.

This approach requires a meta-training phase over a large collection of tasks. If such a collection is available, the meta-learned router can achieve very rapid adaptation to new tasks with only a handful of examples. The connection to few-shot learning is direct: meta-learning the router is essentially learning to recognize which combination of prior capabilities a new task needs, which is precisely the kind of rapid task understanding that few-shot learners are trained to develop.

Modern large language models have introduced a different flavor of architecture-based continual learning: rather than growing the full network, you grow a small set of lightweight adapters attached to a frozen backbone. This approach, sometimes called parameter-efficient continual learning, dramatically reduces the parameter overhead per task while preserving the core representations in the frozen backbone.

Adapter modules, originally introduced for parameter-efficient fine-tuning, insert small bottleneck layers into a frozen pre-trained network. Each adapter consists of a down-projection to a low-dimensional bottleneck, a non-linearity, and an up-projection back to the original dimension:

where:

• $\mathbf{h}$: the hidden state from the frozen backbone at that layer
• $W_{\text{down}} \in \mathbb{R}^{r \times d}$: the down-projection to bottleneck dimension $r$
• $W_{\text{up}} \in \mathbb{R}^{d \times r}$: the up-projection back to dimension $d$
• $\sigma$: a non-linearity (typically ReLU or GELU)
• The residual connection $+\mathbf{h}$ ensures that the adapter starts as an identity transformation when initialized near zero

In the continual learning setting, each new task gets its own set of adapter modules while the backbone remains frozen. This provides the same zero-forgetting guarantee as progressive networks but with a much smaller per-task footprint. If the adapter bottleneck dimension is $r$ and the backbone has $d$-dimensional hidden states, the adapter adds $2rd$ parameters per layer per task instead of the full $d^2$ parameters of a new column. For large transformers where $d$ might be 4096 and $r$ might be 64, this is a 64-fold reduction in per-task overhead.

Low-Rank Adaptation (LoRA), which we discussed in the Parameter-Efficient Fine-Tuning chapter, decomposes weight updates into low-rank matrices. In the continual learning context, each task trains a set of LoRA adapters while the backbone is frozen. The forward pass for task $t$ at a given weight matrix $W_0$ computes:

where:

• $W_0 \in \mathbb{R}^{d \times k}$: the frozen pre-trained weight matrix
• $A^{(t)} \in \mathbb{R}^{r \times k}$: the low-rank down-projection for task $t$, initialized randomly
• $B^{(t)} \in \mathbb{R}^{d \times r}$: the low-rank up-projection for task $t$, initialized to zero
• $r$: the rank, controlling expressiveness versus parameter count
• $\alpha$: a scaling hyperparameter

Because $B^{(t)}$ is initialized to zero, the LoRA adapter starts as an identity operation, and the model for task $t$ begins as the frozen pre-trained model. This initialization property makes it easy to deploy the LoRA approach in sequence: each new task starts with the pre-trained model's capabilities and adds just the task-specific delta represented by the low-rank factors.

The key advantage of LoRA-based continual learning is that the stored per-task parameters can be merged with the backbone for inference. If you know the task identity, you add $B^{(t)} A^{(t)}$ to the backbone weights and run inference without any overhead compared to the base model. If you need to switch tasks frequently, you can maintain a library of low-rank deltas and apply them on demand. The storage footprint for the full task library is the backbone (shared) plus $K$ sets of LoRA matrices (one per task), which is far more compact than $K$ full columns.

One limitation of simply training independent LoRA adapters per task is that the adapters can still interfere if they are applied to a shared backbone that evolves. When all prior tasks use the same frozen backbone, there is no interference. But in settings where the backbone itself is allowed to update (for improved performance on recent tasks), the LoRA adapters from prior tasks may no longer function as expected because the backbone they were calibrated against has shifted.

Orthogonal Low-Rank Adaptation (O-LoRA) addresses this by constraining updates to the backbone to lie in subspaces orthogonal to those used by prior task adapters. Specifically, the gradient of the backbone weights is projected to be orthogonal to the span of prior tasks' adapter matrices. This prevents new backbone updates from disrupting the directions that prior adapters rely on, giving a softer version of the hard-freezing guarantee without completely preventing backbone adaptation.

Let us walk through a concrete implementation of a simplified progressive network to see how the column structure and lateral connections work in practice. We will train on two sequential tasks (two different classification problems) and verify that performance on the first task is perfectly preserved after adding a second column for the second task.

The implementation has three parts. First, we define the ProgressiveColumn class, which represents a single column of the network including its main feed-forward layers and its lateral adapter modules that receive signals from prior columns. Second, we define the ProgressiveNetwork class that manages the list of columns, handles the freezing of prior columns when a new task arrives, and routes inputs to the appropriate column at inference time. Third, we train on two sequential tasks and measure Task 1 accuracy before and after Task 2 training to confirm the structural zero-forgetting guarantee.

The zero forgetting result is structural, not an artifact of the particular dataset or training process. Because column 0's weights are frozen before column 1 is trained, the computation graph for task 0 inputs is completely unchanged. The column-freezing mechanism provides an exact guarantee rather than just a regularization tendency.

Notice also that Task 1's accuracy before and after Task 2 training should match to many decimal places, not just approximately. Any floating-point discrepancy is purely numerical noise rather than forgetting. This is the defining property of progressive networks and distinguishes them from regularization-based methods where forgetting is reduced but never completely eliminated.

Now let us implement the mask-based protection mechanism from HAT. This gives us a fixed-capacity network where each task learns a sparse attention mask over units, and the accumulated masks protect previously claimed capacity from future modification. The implementation has three key components: the task embedding that parameterizes the mask, the annealing schedule that hardens the mask over training, and the gradient zeroing that enforces protection at each update step.

The cumulative mask grows as more tasks are added. Each task carves out a portion of the hidden layer's units, and those units are subsequently protected from modification. The remaining unclaimed units are available for future tasks. When the unclaimed region shrinks to a very small number of units, the network will struggle to accommodate new tasks effectively.

Notice that unlike the progressive network, HAT's protection is not perfect in this simplified implementation. In the full HAT paper, gradient protection is applied to all weight tensors connected to protected neurons, including the output layer and all lateral connections. Our simplified version focuses on fc1 to illustrate the principle.

Architecture-based continual learning involves a richer set of hyperparameters than regularization or replay methods, because you are making decisions about the training objective and the network's physical structure. Understanding which parameters control which trade-offs is essential for applying these methods effectively.

The key parameters for architecture-based continual learning methods are:

• hidden_dim: The total width of the shared network backbone. Larger hidden layers provide more capacity for mask-based methods like HAT to allocate across tasks. For progressive networks, this controls the per-column capacity. Increasing width helps both with per-task accuracy and with the number of tasks that can fit, but at the cost of a larger total model size.
• lateral connections: In progressive networks, the lateral weight matrices are initialized to zero so each column begins independent. Non-zero initialization would allow immediate feature borrowing but risks early-task representations dominating the new column before it has established its own features. The lateral connection rank (if using low-rank adapters) controls the expressivity of cross-column communication.
• s_max: The maximum temperature for HAT mask annealing. Higher values produce harder (more binary) masks but can make gradient descent more difficult near the end of training as the soft mask values saturate. Values in the range 50 to 400 have been reported in practice, with the optimal choice depending on network size and task complexity.
• lam (lambda): The sparsity penalty coefficient in HAT. Higher values force sparser masks, preserving more capacity for future tasks at the expense of potentially using fewer neurons for the current task and reducing per-task accuracy. This parameter effectively controls the future-versus-present trade-off: a high lambda is more conservative about current resource consumption.
• pruning_rate (for PackNet-style methods): The fraction of weights freed after each task. Higher rates free more capacity for future tasks but give each task fewer parameters to work with, harming accuracy on tasks trained later in the sequence. The optimal pruning rate is a function of the total number of tasks anticipated, which must often be estimated in advance.
• expansion threshold (for DEN-style methods): The loss threshold above which the network adds new neurons. A lower threshold triggers expansion more aggressively, giving better per-task accuracy but growing the network more quickly. Setting this threshold requires understanding the typical loss values on convergence for tasks of the expected difficulty.
• adapter rank (for LoRA and adapter methods): Controls the expressiveness of the task-specific delta. Larger rank allows more complex task adaptations but increases per-task storage. Typical values range from 4 to 128 depending on model size and task complexity.

Architecture methods offer the strongest theoretical guarantees against catastrophic forgetting, but they come with practical constraints that limit their applicability in the general continual learning setting.

The most fundamental limitation is the assumption of task identity. Nearly every architecture method described in this chapter, from progressive networks to HAT to PackNet, requires knowing the task ID at inference time to route the input to the correct column, mask, or module subset. This is a strong assumption. In realistic deployment scenarios, the system may encounter inputs without any indication of which task they belong to. Methods that relax this requirement, such as using a meta-learned router trained to infer task membership from input statistics, address this but add their own complexity and error modes.

The second major limitation is capacity management. Fixed-capacity methods like HAT and PackNet eventually run out of room as the number of tasks grows. Each task consumes some fraction of the total available capacity, and the remaining capacity for future tasks shrinks monotonically. There is no free lunch: you can either accept growing storage (progressive networks) or accept diminishing per-task capacity (fixed-capacity methods). The right trade-off depends on the application; a robot that learns tens of manipulation tasks can afford progressive growth, but a production model that must learn from a continuous stream of user interactions cannot. Some hybrid methods try to get the best of both by starting with a fixed capacity and expanding only when a low-water threshold is crossed, but this requires careful monitoring and can still result in either excessive growth or premature degradation.

The third limitation is that most architecture methods assume a clear boundary between tasks. The system needs to know when one task ends and the next begins so it can freeze columns, update cumulative masks, or route to a new set of adapters. In practice, task boundaries are often unclear, especially in settings like continual pre-training of language models where the data distribution shifts gradually rather than abruptly. Detecting these shifts and deciding when to add new capacity or update protection masks is itself a non-trivial problem that adds to the complexity of deploying architecture-based methods.

The impact of architecture methods on the broader field has been substantial despite these limitations. Progressive networks demonstrated that continual learning without any forgetting is architecturally achievable, which was an important proof of concept even if the linear growth was impractical. HAT's mask-based approach influenced later work on network pruning and sparse subnetworks, connecting continual learning to efficiency research and establishing a clean link between sparsity regularization and memory protection. PathNet's evolutionary routing was an early demonstration that structural modularity could emerge from learning rather than being hand-designed, inspiring later differentiable routing methods. The adapter-based and LoRA-based approaches that dominate modern continual fine-tuning of large language models are direct descendants of the architectural thinking developed in this line of work.

Looking ahead, the most promising direction is the combination of architecture methods with other continual learning strategies. A system that uses small modular adapters (architecture) combined with selective replay of critical examples (replay methods, covered in the previous chapter) and regularization on the shared backbone (regularization methods) can potentially address all three failure modes: catastrophic forgetting of old tasks, inadequate capacity for new tasks, and inefficient reuse of previously learned representations. The convergence of these strategies in the era of large pre-trained models, where the backbone is already rich with learned representations and the per-task delta can be very small, suggests that the practical barriers to scalable continual learning are lower now than they have ever been.

Architecture methods address catastrophic forgetting by modifying the network structure rather than the training objective. The key insight is that forgetting occurs when parameters are shared and overwritten across tasks, so the remedy is to either prevent overwriting through freezing (progressive networks, HAT, PackNet) or allocate separate capacity through structural expansion (DEN) or modular composition (PathNet, MoE).

The main trade-offs to keep in mind are:

• Progressive networks freeze entire columns per task, guaranteeing zero forgetting but requiring linear parameter growth. Lateral connections provide automatic forward transfer based on what the new column finds useful from prior columns.
• PackNet fits multiple tasks into a fixed network through pruning and masking, but each successive task receives fewer parameters, creating a hard capacity ceiling that depends on the pruning rate and the total number of tasks.
• HAT learns binary attention masks per task to protect claimed neurons, with a sparsity penalty encouraging sparse masks that preserve room for future tasks. The temperature annealing strategy allows smooth gradient-based learning while converging to hard protection boundaries.
• DEN dynamically expands the network when existing capacity proves insufficient, adding neurons where they are most needed and connecting them to existing structure rather than initializing randomly.
• Modular approaches maintain a library of reusable components and represent each task as a composition, enabling knowledge sharing when tasks overlap in structure. PathNet uses evolutionary search over module assignments, while MoE-based methods use learned routers that can be trained end-to-end.
• Adapter and LoRA methods attach small task-specific parameter sets to a frozen backbone, combining the zero-forgetting guarantee of column freezing with the parameter efficiency needed to scale to large pre-trained models and many tasks.

Architecture methods work best when the number of tasks is bounded and task identities are available at inference time. When these conditions do not hold, they must be combined with task inference mechanisms or other continual learning strategies. The next chapter on Continual Learning Evaluation will examine how to measure the quality of these approaches, including the forward transfer and backward transfer metrics that capture whether new tasks benefit from old knowledge and whether old tasks are preserved after new learning.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about architecture methods for continual learning.