Architecture Methods › Modular Networks

Modular architectures take the most compositional view of architecture methods. Instead of a single monolithic network, the system maintains a library of reusable modules, and each task is represented as a particular combination or composition of those modules. The intuition is that tasks often share sub-skills: translation and summarization both require understanding sentence structure; sentiment analysis and topic classification both require word-level representations. If these sub-skills live in separate modules, new tasks that share sub-skills with old ones can reuse those modules without modification.

The modular hypothesis is intellectually appealing because it mirrors how human cognition is described in cognitive science. People are generally understood to have domain-specific mental modules that handle particular types of input (faces, language, spatial reasoning) and that can be combined flexibly for novel problem-solving. Whether artificial neural networks trained from data naturally develop anything like this structure is an open empirical question, but modular architectures try to impose it structurally rather than waiting for it to emerge.

A natural modular architecture is a mixture of experts (MoE), where each task selects a weighted combination of expert networks for its forward pass. As we covered in the Mixture of Experts chapter, each expert is a small network (often a feed-forward layer or a small block), and a router assigns input-dependent weights to determine how much each expert contributes to the output. In the continual learning setting, existing experts are frozen when new tasks arrive and new experts are added as needed. The router for each new task is trained to select appropriate combinations of old experts (for knowledge transfer) and new experts (for task-specific adaptation). The loss for task $t$ with an MoE backbone is:

$$\mathcal{L}_{\text{MoE}}^{(t)} = \mathcal{L}_{\text{task}}^{(t)}\!\left( \sum_{k=1}^{K} g_k^{(t)}(\mathbf{x}) \cdot f_k(\mathbf{x}) \right)$$

where:

• $K$: total number of experts (frozen old plus new)
• $g_k^{(t)}(\mathbf{x})$: the router weight assigned to expert $k$ for task $t$ on input $\mathbf{x}$
• $f_k(\mathbf{x})$: the output of expert $k$ on input $\mathbf{x}$
• The weights $g_k^{(t)}$ satisfy $\sum_k g_k^{(t)} = 1$ and are non-negative

The frozen experts from earlier tasks contribute to the new task's predictions in proportion to how useful the router finds them, enabling automatic knowledge transfer without any explicit specification of which old skills are relevant.

One important design question is whether the experts should be frozen completely or allowed to specialize through lightweight adaptation. A common variant uses a small set of fully trainable base experts shared across all tasks and a set of task-specific delta experts that learn residual adjustments. This mirrors the LoRA-style approach to parameter-efficient fine-tuning, where a frozen base model is augmented by low-rank adapters. The advantage is that the base experts can improve as more tasks expose them to diverse data, while the delta experts ensure each task can still express task-specific patterns.

PathNet (Fernando et al., 2017) is an early modular architecture that explicitly represents the network as a directed graph of reusable modules. The network is a grid of $L$ layers, each with $M$ modules. At each layer, the active path uses at most $N \leq M$ modules. A genetic algorithm evolves a population of candidate path configurations, and each candidate path is trained for a few gradient steps. The best-performing path is selected, its modules are frozen, and the process repeats for the next task.

The module output at layer $l$ along path $\mathcal{P}$ is:

$$h_l = \text{ReLU}\!\left( \sum_{j \in \mathcal{P}_l} W_j h_{l-1} \right)$$

where $\mathcal{P}_l \subseteq \{1, \ldots, M\}$ is the set of modules active at layer $l$ along path $\mathcal{P}$.

PathNet enables precise freezing: once a path is selected for a task, those specific modules (not the entire network) are frozen. Modules not on the winning path for any task remain plastic and can be recruited by future tasks. This allows parameter sharing when two tasks select overlapping paths, unlike progressive networks where each task has exclusive columns.

The evolutionary search is computationally expensive, which is PathNet's main practical limitation. For large networks or many tasks, the search itself becomes a bottleneck. Running a genetic algorithm across hundreds of candidate paths, each requiring several gradient steps to evaluate, imposes significant computational overhead that is hard to justify in applications where training speed matters. Later work replaced the evolutionary search with differentiable routing learned through gradient descent, which is both faster and more compatible with standard deep learning infrastructure.

An appealing property of PathNet is that you can visualize which modules were selected for which tasks, producing an interpretable map of knowledge sharing. If tasks A and B share many modules, the learned paths tell you directly that the system found structural overlap between them. This transparency is harder to achieve with mask-based methods where the shared knowledge is encoded in the same weight values rather than in distinct module assignments.

A more principled approach to module composition is meta-learning the routing mechanism itself. Rather than treating the composition as a static assignment (as in PathNet) or a fixed-capacity mixture (as in MoE), meta-learning methods learn how to assemble modules based on task descriptions or small amounts of task data.

One formulation uses an amortized inference network that takes a set of labeled examples from the new task as input and outputs module selection probabilities. This is trained across a distribution of tasks such that the inferred composition generalizes to held-out tasks. The module library is updated between meta-training episodes to ensure it contains modules worth composing.

This approach requires a meta-training phase over a large collection of tasks. If such a collection is available, the meta-learned router can achieve very rapid adaptation to new tasks with only a handful of examples. The connection to few-shot learning is direct: meta-learning the router is essentially learning to recognize which combination of prior capabilities a new task needs, which is precisely the kind of rapid task understanding that few-shot learners are trained to develop.