Architecture Methods › Summary

Architecture methods address catastrophic forgetting by modifying the network structure rather than the training objective. The key insight is that forgetting occurs when parameters are shared and overwritten across tasks, so the remedy is to either prevent overwriting through freezing (progressive networks, HAT, PackNet) or allocate separate capacity through structural expansion (DEN) or modular composition (PathNet, MoE).

The main trade-offs to keep in mind are:

• Progressive networks freeze entire columns per task, guaranteeing zero forgetting but requiring linear parameter growth. Lateral connections provide automatic forward transfer based on what the new column finds useful from prior columns.
• PackNet fits multiple tasks into a fixed network through pruning and masking, but each successive task receives fewer parameters, creating a hard capacity ceiling that depends on the pruning rate and the total number of tasks.
• HAT learns binary attention masks per task to protect claimed neurons, with a sparsity penalty encouraging sparse masks that preserve room for future tasks. The temperature annealing strategy allows smooth gradient-based learning while converging to hard protection boundaries.
• DEN dynamically expands the network when existing capacity proves insufficient, adding neurons where they are most needed and connecting them to existing structure rather than initializing randomly.
• Modular approaches maintain a library of reusable components and represent each task as a composition, enabling knowledge sharing when tasks overlap in structure. PathNet uses evolutionary search over module assignments, while MoE-based methods use learned routers that can be trained end-to-end.
• Adapter and LoRA methods attach small task-specific parameter sets to a frozen backbone, combining the zero-forgetting guarantee of column freezing with the parameter efficiency needed to scale to large pre-trained models and many tasks.

Architecture methods work best when the number of tasks is bounded and task identities are available at inference time. When these conditions do not hold, they must be combined with task inference mechanisms or other continual learning strategies. The next chapter on Continual Learning Evaluation will examine how to measure the quality of these approaches, including the forward transfer and backward transfer metrics that capture whether new tasks benefit from old knowledge and whether old tasks are preserved after new learning.