Architecture Methods › Limitations and Impact

Architecture methods offer the strongest theoretical guarantees against catastrophic forgetting, but they come with practical constraints that limit their applicability in the general continual learning setting.

The most fundamental limitation is the assumption of task identity. Nearly every architecture method described in this chapter, from progressive networks to HAT to PackNet, requires knowing the task ID at inference time to route the input to the correct column, mask, or module subset. This is a strong assumption. In realistic deployment scenarios, the system may encounter inputs without any indication of which task they belong to. Methods that relax this requirement, such as using a meta-learned router trained to infer task membership from input statistics, address this but add their own complexity and error modes.

The second major limitation is capacity management. Fixed-capacity methods like HAT and PackNet eventually run out of room as the number of tasks grows. Each task consumes some fraction of the total available capacity, and the remaining capacity for future tasks shrinks monotonically. There is no free lunch: you can either accept growing storage (progressive networks) or accept diminishing per-task capacity (fixed-capacity methods). The right trade-off depends on the application; a robot that learns tens of manipulation tasks can afford progressive growth, but a production model that must learn from a continuous stream of user interactions cannot. Some hybrid methods try to get the best of both by starting with a fixed capacity and expanding only when a low-water threshold is crossed, but this requires careful monitoring and can still result in either excessive growth or premature degradation.

The third limitation is that most architecture methods assume a clear boundary between tasks. The system needs to know when one task ends and the next begins so it can freeze columns, update cumulative masks, or route to a new set of adapters. In practice, task boundaries are often unclear, especially in settings like continual pre-training of language models where the data distribution shifts gradually rather than abruptly. Detecting these shifts and deciding when to add new capacity or update protection masks is itself a non-trivial problem that adds to the complexity of deploying architecture-based methods.

The impact of architecture methods on the broader field has been substantial despite these limitations. Progressive networks demonstrated that continual learning without any forgetting is architecturally achievable, which was an important proof of concept even if the linear growth was impractical. HAT's mask-based approach influenced later work on network pruning and sparse subnetworks, connecting continual learning to efficiency research and establishing a clean link between sparsity regularization and memory protection. PathNet's evolutionary routing was an early demonstration that structural modularity could emerge from learning rather than being hand-designed, inspiring later differentiable routing methods. The adapter-based and LoRA-based approaches that dominate modern continual fine-tuning of large language models are direct descendants of the architectural thinking developed in this line of work.

Looking ahead, the most promising direction is the combination of architecture methods with other continual learning strategies. A system that uses small modular adapters (architecture) combined with selective replay of critical examples (replay methods, covered in the previous chapter) and regularization on the shared backbone (regularization methods) can potentially address all three failure modes: catastrophic forgetting of old tasks, inadequate capacity for new tasks, and inefficient reuse of previously learned representations. The convergence of these strategies in the era of large pre-trained models, where the backbone is already rich with learned representations and the per-task delta can be very small, suggests that the practical barriers to scalable continual learning are lower now than they have ever been.