Architecture Methods › Architecture Methods: Progressive Networks, Expert Expansion, and Modular Approaches

Regularization and replay are two well-studied answers to catastrophic forgetting: the first constrains how much existing weights can move, and the second makes old data available during new training. Both approaches share a fundamental assumption, namely that a single fixed-capacity network will hold all current and future knowledge. Architecture methods reject that assumption. They ask instead: what if the network itself changes as new tasks arrive?

The core insight is that catastrophic forgetting occurs because learning a new task overwrites the parameters used by old ones. If you can quarantine old parameters, or expand the network to accommodate new tasks without disturbing existing representations, forgetting becomes a structural non-issue rather than an optimization challenge. As we discussed in the Regularization Methods chapter, Elastic Weight Consolidation identifies which weights matter most to old tasks and penalizes their movement. Architecture methods take a more radical route by either freezing those weights entirely or routing new tasks through fresh capacity altogether.

Think of the problem in purely structural terms. A single network is like a shared whiteboard. Every time you write something new on it, you risk erasing old content. Regularization gives each piece of content a varying degree of erasure protection. Replay keeps copies of old content nearby so you can restore it if it gets smudged. Architecture methods instead give each piece of content its own dedicated section of the board, or expand the board whenever it fills up. This analogy breaks down in detail, but it captures the philosophical difference: architecture methods solve the forgetting problem spatially rather than temporally.

This chapter covers four families of architecture-based approaches. Progressive networks add lateral connections and new columns for each task, keeping old columns frozen. Expansion methods dynamically grow the network capacity when they detect insufficient resources. Mask-based methods like HAT (Hard Attention to the Task) learn binary attention patterns that protect previously claimed units while working within a fixed total parameter budget. Modular approaches decompose knowledge into reusable components that different tasks can assemble in different combinations. Each family makes different trade-offs between memory growth, forward transfer, and backward transfer.