Regularization Methods › The Regularization Approach to Catastrophic Forgetting

Before diving into specific algorithms, it helps to understand the general structure of regularization-based continual learning. When you train a neural network on a sequence of tasks $T_1, T_2, \ldots, T_N$, catastrophic forgetting occurs because each task's training loss treats all parameters as equally adjustable. The optimizer does not know that some parameters have already been carefully tuned for previous tasks.

Think of it this way. After training on task A, the model has converged to some parameter configuration $\theta^*_A$ that works well for that task. The loss landscape around $\theta^*_A$ is curved: in some directions, moving away from $\theta^*_A$ causes the task-A loss to rise steeply (high curvature), while in other directions the loss rises only gradually (low curvature). Parameters lying in steep directions are critical to task A performance; those in flat directions barely matter.

When you then train on task B, the optimizer follows the gradient of the task-B loss, moving $\theta$ away from $\theta^*_A$ toward a new optimum $\theta^*_B$. If the parameter most critical for task A happens to lie in a direction that the optimizer wants to change dramatically for task B, the result is severe forgetting. Regularization methods prevent this by adding a term to the task-B loss that penalizes moving away from $\theta^*_A$ in those critical directions. The penalty is small in flat directions (where moving is safe) and large in steep directions (where moving destroys task-A performance). The result is that the optimizer must find a compromise: it cannot move as freely as it would without the penalty, but it can still find solutions that work reasonably well for task B without fully sacrificing task A.

The regularization approach adds an auxiliary loss term that penalizes changes to parameters important for past tasks. Given a model with parameters $\theta$, the total loss when training on task $T_k$ becomes:

where:

• $\mathcal{L}_{T_k}(\theta)$: the task-specific loss (cross-entropy, MSE, etc.) for task $T_k$
• $\mathcal{L}_{\text{reg}}(\theta, \theta^*_{k-1})$: a regularization term that penalizes deviation from the previous optimal parameters $\theta^*_{k-1}$
• $\lambda$: a scalar hyperparameter controlling the strength of the regularization
• $\theta^*_{k-1}$: the parameter values after training on all tasks up to and including $T_{k-1}$

The key difference between methods lies in how $\mathcal{L}_{\text{reg}}$ is defined. The simplest possible choice is $L_2$ regularization: penalize the squared Euclidean distance between current and previous parameters uniformly. But this treats all parameters equally, which is exactly the wrong approach. A parameter that does not matter for previous tasks should be free to change; one that is critical should be heavily protected. Uniform $L_2$ regularization would hamstring the optimizer by constraining even the most redundant weights, reducing the model's capacity to learn new tasks without any corresponding benefit for old ones.

This motivates a weighted regularization term:

where:

• $\theta_i$: the current value of the $i$-th parameter
• $\theta^*_i$: the value of the $i$-th parameter after training on previous tasks
• $\Omega_i$: the importance weight for parameter $i$, indicating how critical it is for past tasks

The central question becomes: how do you compute $\Omega_i$? Different regularization methods provide different answers, each with their own computational tradeoffs and theoretical justifications. The weighted form above is the shared skeleton; EWC and SI are two different ways to fill in the importance weights.

One way to think about importance is through the lens of loss sensitivity. A parameter is important if the task-A loss is sensitive to changes in it. Mathematically, this sensitivity is related to the second derivative of the loss with respect to that parameter. If the Hessian is large in the direction of parameter $i$, then even small deviations from $\theta^*_{A,i}$ will cause a large increase in $\mathcal{L}_A$. If the Hessian is small, the loss is flat in that direction and deviations are tolerable. EWC formalizes this intuition using the Fisher information matrix. SI takes a different route, looking not at the geometry of the loss surface at convergence but at the dynamics of how the parameter moved during training.