Architecture Methods › Expansion Methods

Rather than adding a complete new column per task, expansion methods aim to grow the network more surgically. The core idea is to detect when an existing network lacks sufficient capacity for a new task and add capacity exactly where it is needed, leaving intact the parts that are already adequate.

The philosophical motivation here is efficiency. Progressive networks treat every new task as equally demanding of full new capacity, regardless of whether the new task is nearly identical to a prior one (requiring very little new capacity) or completely novel (requiring a full complement of new parameters). Expansion methods try to adaptively allocate capacity based on actual need.

PackNet (Mallya and Lazebnik, 2018) uses iterative pruning and weight freezing to fit multiple tasks into a single network. The lifecycle for each new task proceeds in three phases.

First, the network is trained on the new task to convergence using all available weights. This training uses whatever capacity has not been frozen by prior tasks. Second, a fraction of weights with the smallest absolute value are pruned (zeroed out) to free capacity for future tasks. Pruning by magnitude is a simple but effective heuristic: weights close to zero contribute little to the model's output, so removing them causes minimal performance degradation. Third, the remaining non-zero weights for this task are frozen, protecting them from future modification.

Future tasks can only use the pruned (freed) weights, which are initially random and available for learning. This process repeats for each new task in sequence, and the final network has a fixed total size regardless of the number of tasks.

The appeal of PackNet is that the total parameter count is constant. There is no memory growth as tasks accumulate. The cost is that each successive task has fewer available parameters, since the frozen weights from all prior tasks are unavailable. The capacity allocated to task $k$ is roughly $(1 - \rho)^{k-1}$ times the full network size, where $\rho$ is the pruning fraction per task. For a network with 1 million parameters and a pruning rate of 50%, task 10 has access to only $(0.5)^9 \approx 0.2\%$ of the network, which is severely limited. Choosing a small pruning rate (say 10%) extends the useful life of the network but reserves less capacity per task.

PackNet still requires task identities at inference time to know which subset of weights to use. It applies a binary mask for each task: the forward pass multiplies all weights by the task-specific mask, effectively silencing weights assigned to other tasks. The masks are stored as binary vectors, so their memory overhead is small compared to the weights themselves.

One practical detail is the order of pruning. If you prune aggressively early (to reserve capacity for many future tasks), each task gets abundant room but the early-task models are smaller and potentially less accurate. If you prune conservatively, early tasks are accurate but fewer tasks can fit. This trade-off requires knowing in advance how many tasks the system will encounter, which is not always possible.

A more principled expansion approach is to let the network detect when it needs more capacity and grow accordingly. DEN (Lifelong Learning with Dynamically Expandable Networks, Yoon et al. 2018) uses selective retraining and dynamic expansion together.

When a new task arrives, DEN first identifies which neurons in the network are most relevant to the new task by measuring activation patterns on a small set of new-task samples. It retrains only those relevant neurons, leaving others frozen. If the loss on new-task data remains above a threshold after selective retraining, DEN adds new neurons to accommodate the remaining difficulty. Neurons that are similar (by cosine similarity of weights) to existing neurons are connected to the existing structure rather than initialized randomly, encouraging reuse of learned feature detectors.

The key equations for neuron expansion in DEN involve a reconstruction loss that combines task performance with a regularization term penalizing drift from prior neuron activations. For layer $l$, the loss when expanding is:

$$\mathcal{L}_{\text{expand}}^{(t)} = \mathcal{L}_{\text{task}}^{(t)} + \mu \sum_{j \in \mathcal{S}} \left\| h_j^l - \hat{h}_j^l \right\|^2$$

where:

• $\mathcal{L}_{\text{task}}^{(t)}$: the supervised loss on the new task $t$
• $\mathcal{S}$: the set of neurons selected for retraining (not newly added ones)
• $h_j^l$: current activation of neuron $j$ at layer $l$
• $\hat{h}_j^l$: reference activation of neuron $j$ before retraining on task $t$
• $\mu$: drift penalty coefficient controlling how much neuron activations are allowed to change

This formulation allows selected neurons to adapt to the new task while discouraging them from drifting far from the representations that made them useful on old tasks. Newly added neurons are unrestricted and can learn new features freely.

The appeal of DEN over simple column expansion is that it is parsimonious. A new task that is very similar to prior tasks may not trigger any expansion at all, relying entirely on existing neurons with selective fine-tuning. Only tasks that cannot be represented by existing capacity cause the network to grow. This means DEN's memory footprint depends on the actual novelty of the task sequence rather than growing unconditionally.

The challenge with DEN is that it has more hyperparameters to set: the threshold for deciding that selective retraining is insufficient, the number of neurons to add when expansion is triggered, and the drift coefficient $\mu$. Getting these right in practice often requires task-specific tuning, and the selective neuron identification step adds computational overhead at the start of each new task.