Regularization Methods › Regularization Methods

In the previous chapter on the Continual Learning Problem, we saw how neural networks suffer from catastrophic forgetting: when a model learns new tasks sequentially, the gradient updates for the new task overwrite the weights that mattered for previous tasks. The network has no memory of what it once knew. The optimizer is a relentless hill-climber that follows the gradient of the current task's loss, and it has no mechanism to distinguish between weights that are critical to past performance and weights that are freely available for repurposing. Everything is equally up for grabs, and so everything tends to change.

Regularization methods for continual learning address this problem not by storing old data, but by protecting the parameters that were most important for past tasks from being changed too much. The core philosophy is surgical constraint: identify which parameters encode critical knowledge for task A, then penalize large changes to those parameters when training on task B. The new task's gradients are still applied, but they are constrained in directions that would damage previously acquired knowledge. This creates a form of soft memory: the model can adapt to new tasks while retaining the essence of what it previously learned.

This philosophy is elegant because it imposes no architectural overhead. You do not need to add new network branches, expand the model, or maintain a separate replay buffer. The same parameter vector carries knowledge of all tasks simultaneously, distinguished only by which parameters happen to encode which task's representations. If tasks have compatible representations, this shared parameter space is sufficient. If they do not, the limitations of this approach become apparent, and you will need to look toward architecture-based or replay-based methods instead.

This chapter covers two foundational regularization methods: Elastic Weight Consolidation (EWC) and Synaptic Intelligence (SI). Both follow the same general philosophy but differ in how they measure parameter importance and when that measurement is computed. Understanding these methods will give you the conceptual tools to reason about any regularization-based approach to continual learning, including the methods we will encounter when discussing architecture methods and evaluation in upcoming chapters.