Regularization Methods: EWC and Synaptic Intelligence

In the previous chapter on the Continual Learning Problem, we saw how neural networks suffer from catastrophic forgetting: when a model learns new tasks sequentially, the gradient updates for the new task overwrite the weights that mattered for previous tasks. The network has no memory of what it once knew. The optimizer is a relentless hill-climber that follows the gradient of the current task's loss, and it has no mechanism to distinguish between weights that are critical to past performance and weights that are freely available for repurposing. Everything is equally up for grabs, and so everything tends to change.

Regularization methods for continual learning address this problem not by storing old data, but by protecting the parameters that were most important for past tasks from being changed too much. The core philosophy is surgical constraint: identify which parameters encode critical knowledge for task A, then penalize large changes to those parameters when training on task B. The new task's gradients are still applied, but they are constrained in directions that would damage previously acquired knowledge. This creates a form of soft memory: the model can adapt to new tasks while retaining the essence of what it previously learned.

This philosophy is elegant because it imposes no architectural overhead. You do not need to add new network branches, expand the model, or maintain a separate replay buffer. The same parameter vector carries knowledge of all tasks simultaneously, distinguished only by which parameters happen to encode which task's representations. If tasks have compatible representations, this shared parameter space is sufficient. If they do not, the limitations of this approach become apparent, and you will need to look toward architecture-based or replay-based methods instead.

This chapter covers two foundational regularization methods: Elastic Weight Consolidation (EWC) and Synaptic Intelligence (SI). Both follow the same general philosophy but differ in how they measure parameter importance and when that measurement is computed. Understanding these methods will give you the conceptual tools to reason about any regularization-based approach to continual learning, including the methods we will encounter when discussing architecture methods and evaluation in upcoming chapters.

Before diving into specific algorithms, it helps to understand the general structure of regularization-based continual learning. When you train a neural network on a sequence of tasks $T_1, T_2, \ldots, T_N$, catastrophic forgetting occurs because each task's training loss treats all parameters as equally adjustable. The optimizer does not know that some parameters have already been carefully tuned for previous tasks.

Think of it this way. After training on task A, the model has converged to some parameter configuration $\theta^*_A$ that works well for that task. The loss landscape around $\theta^*_A$ is curved: in some directions, moving away from $\theta^*_A$ causes the task-A loss to rise steeply (high curvature), while in other directions the loss rises only gradually (low curvature). Parameters lying in steep directions are critical to task A performance; those in flat directions barely matter.

When you then train on task B, the optimizer follows the gradient of the task-B loss, moving $\theta$ away from $\theta^*_A$ toward a new optimum $\theta^*_B$. If the parameter most critical for task A happens to lie in a direction that the optimizer wants to change dramatically for task B, the result is severe forgetting. Regularization methods prevent this by adding a term to the task-B loss that penalizes moving away from $\theta^*_A$ in those critical directions. The penalty is small in flat directions (where moving is safe) and large in steep directions (where moving destroys task-A performance). The result is that the optimizer must find a compromise: it cannot move as freely as it would without the penalty, but it can still find solutions that work reasonably well for task B without fully sacrificing task A.

The regularization approach adds an auxiliary loss term that penalizes changes to parameters important for past tasks. Given a model with parameters $\theta$, the total loss when training on task $T_k$ becomes:

where:

• $\mathcal{L}_{T_k}(\theta)$: the task-specific loss (cross-entropy, MSE, etc.) for task $T_k$
• $\mathcal{L}_{\text{reg}}(\theta, \theta^*_{k-1})$: a regularization term that penalizes deviation from the previous optimal parameters $\theta^*_{k-1}$
• $\lambda$: a scalar hyperparameter controlling the strength of the regularization
• $\theta^*_{k-1}$: the parameter values after training on all tasks up to and including $T_{k-1}$

The key difference between methods lies in how $\mathcal{L}_{\text{reg}}$ is defined. The simplest possible choice is $L_2$ regularization: penalize the squared Euclidean distance between current and previous parameters uniformly. But this treats all parameters equally, which is exactly the wrong approach. A parameter that does not matter for previous tasks should be free to change; one that is critical should be heavily protected. Uniform $L_2$ regularization would hamstring the optimizer by constraining even the most redundant weights, reducing the model's capacity to learn new tasks without any corresponding benefit for old ones.

This motivates a weighted regularization term:

where:

• $\theta_i$: the current value of the $i$-th parameter
• $\theta^*_i$: the value of the $i$-th parameter after training on previous tasks
• $\Omega_i$: the importance weight for parameter $i$, indicating how critical it is for past tasks

The central question becomes: how do you compute $\Omega_i$? Different regularization methods provide different answers, each with their own computational tradeoffs and theoretical justifications. The weighted form above is the shared skeleton; EWC and SI are two different ways to fill in the importance weights.

One way to think about importance is through the lens of loss sensitivity. A parameter is important if the task-A loss is sensitive to changes in it. Mathematically, this sensitivity is related to the second derivative of the loss with respect to that parameter. If the Hessian is large in the direction of parameter $i$, then even small deviations from $\theta^*_{A,i}$ will cause a large increase in $\mathcal{L}_A$. If the Hessian is small, the loss is flat in that direction and deviations are tolerable. EWC formalizes this intuition using the Fisher information matrix. SI takes a different route, looking not at the geometry of the loss surface at convergence but at the dynamics of how the parameter moved during training.

Elastic Weight Consolidation (EWC), introduced by Kirkpatrick et al. (2017), derives its importance weights from the curvature of the loss landscape using the Fisher information matrix. The name comes from an analogy to elasticity: important weights are held in place by strong springs, while unimportant weights can move freely. The spring constant for each weight is proportional to its importance, so the restoring force grows linearly with displacement.

The paper that introduced EWC drew an explicit analogy to how biological neural systems achieve continual learning. Neuroscientists have long observed that synaptic connections in the brain are not equally plastic. Some synapses are stable and resist modification, while others remain highly plastic and readily respond to new experiences. This homeostatic plasticity prevents older, well-established memories from being overwritten by every new learning episode. EWC attempts to replicate this behavior computationally: high-importance parameters are the "stable synapses" and low-importance ones are the "plastic synapses."

To understand EWC's importance measure, we need to think carefully about what determines whether a parameter is important. Intuitively, a parameter is important for task $T$ if small changes to it cause large changes in the task loss. This is exactly what the second derivative of the loss surface captures. But computing the full Hessian of the loss for large neural networks is prohibitively expensive: for a network with $n$ parameters, the Hessian is an $n \times n$ matrix that requires $O(n^2)$ space and expensive computation.

The Fisher information matrix $F$ provides a principled and computationally tractable way to approximate this curvature. For a probabilistic model with parameters $\theta$, the Fisher information matrix is defined as:

where:

• $p(x | \theta)$: the model's output probability distribution over data $x$ given parameters $\theta$
• $\nabla_\theta \log p(x | \theta)$: the gradient of the log-likelihood with respect to parameters (also called the score function)
• The outer product creates a matrix capturing the covariance of the score function
• The expectation is taken over the training data distribution for the previous task

The Fisher matrix is particularly useful because it equals the expected Hessian of the negative log-likelihood under certain regularity conditions. This equality, known as the Fisher information identity, means that the curvature information you want for measuring parameter importance is exactly what the Fisher matrix encodes. The diagonal entries $F_{ii}$ therefore approximate the curvature of the loss in the direction of each parameter: a large $F_{ii}$ means the loss changes rapidly when $\theta_i$ changes, indicating high importance.

Why use the Fisher rather than the Hessian directly? The Fisher is always positive semi-definite (because it is an expectation of outer products), which means it is a valid metric on parameter space. The Hessian can be indefinite away from a local minimum, which creates problems for the regularization interpretation. More importantly, the Fisher can be estimated using only first-order information: you just need gradients, not second-order computations. This makes it computationally accessible through automatic differentiation.

EWC's derivation goes deeper than just using the Fisher as a heuristic. The method has a principled Bayesian justification that reveals why the Fisher is the right thing to use. After training on task $A$, the posterior distribution over parameters is $p(\theta | \mathcal{D}_A)$. When training on task $B$, the ideal Bayesian update would compute:

This equation says: the posterior over parameters given both datasets is proportional to the likelihood of the new data times the prior, where the prior is the posterior from the previous task. In other words, the old posterior plays the role of a prior for learning the new task. This is Bayesian updating, and it is the ideal way to incorporate new evidence without forgetting old evidence.

The challenge is that $p(\theta | \mathcal{D}_A)$ is generally intractable for neural networks. The posterior over a high-dimensional parameter space with a nonlinear model has no closed form, and sampling from it is computationally prohibitive. EWC approximates this posterior using a Laplace approximation: a Gaussian centered at the MAP estimate $\theta^*_A$ with covariance equal to the inverse Fisher information matrix.

The Laplace approximation is the simplest possible Gaussian approximation to a posterior. It treats the log-posterior as a quadratic function near its maximum (the MAP estimate), which is equivalent to fitting a Gaussian whose precision matrix (inverse covariance) equals the curvature of the log-posterior at the mode. Under the assumption that the prior is flat (or wide), the curvature of the log-posterior equals the curvature of the log-likelihood, which is precisely the Fisher information matrix.

Taking the negative log of this Gaussian prior and using it as a regularization term during training on $\mathcal{D}_B$ yields the EWC objective:

where:

• $\mathcal{L}_B(\theta)$: the loss on task $B$'s training data
• $F_{A,ii}$: the $i$-th diagonal entry of the Fisher information matrix computed on task $A$'s data
• $\theta^*_{A,i}$: the $i$-th parameter value at the optimum for task $A$
• $\lambda$: the EWC regularization strength

The diagonal Fisher approximation makes the method computationally tractable. Computing the full Fisher matrix requires $O(|\theta|^2)$ storage, which is infeasible for modern networks with millions of parameters. The diagonal approximation reduces this to $O(|\theta|)$ by keeping only the diagonal entries, treating each parameter independently. This independence assumption discards any correlations between parameters, which is a significant approximation, but one that makes the method scalable while retaining most of the benefit.

The Bayesian framing also clarifies what EWC is doing conceptually. Rather than adding an arbitrary regularizer, EWC performs approximate Bayesian inference. The EWC loss function is the negative log of the approximate posterior given both tasks' data. In this sense, EWC's theoretical grounding is as strong as the Laplace approximation itself, and the main source of error is the diagonal approximation to the Fisher, not the Gaussian approximation to the posterior per se.

In practice, the diagonal Fisher is approximated by averaging the squared gradients over the training data. For each parameter $\theta_i$:

where:

• $\mathcal{D}_A$: the training dataset for task $A$
• $p(y | x, \theta)$: the model's predicted probability for label $y$ given input $x$
• The squared gradient approximates the curvature at the current parameter values

This computation runs after training on task $A$, requiring a forward and backward pass over the data. The result is a vector of the same dimension as $\theta$, where each entry captures the expected squared gradient for that parameter. The implementation is straightforward: run the training data through the network, compute gradients of the log-likelihood with respect to each parameter, square them, and average.

Parameters in early layers (close to the input) tend to have lower Fisher values because small perturbations there must propagate through many nonlinearities before affecting the output. By the time the perturbation reaches the final layer, the nonlinearities have dampened much of the sensitivity. Parameters in final layers often have higher values because they directly shape the prediction. This creates a natural layering of protection: the last few layers of the network are typically the most constrained by EWC, while the earlier representation layers are relatively free to adapt. Whether this pattern matches biological reality is an interesting open question, but it has important practical implications for architecture design.

When training on a third task $T_C$ after having learned $T_A$ and $T_B$, the regularization term must protect parameters important to both prior tasks. There are two common approaches, and the right choice depends on how many tasks you expect and how much memory you can afford.

Online EWC accumulates the Fisher information across tasks:

This approximates the joint Fisher over all previous tasks with a single merged importance vector. The advantage is that memory requirements stay constant: you always store just one importance vector and one set of reference parameters. The disadvantage is that this approximation becomes less accurate as more tasks accumulate. Merging Fisher matrices discards the task-specific anchor points, so the combined importance vector no longer corresponds to any single set of reference parameters. In practice, researchers often maintain a single reference point at the most recent task's optimum and use the accumulated Fisher as the importance measure, accepting some approximation error.

Task-specific EWC maintains separate Fisher matrices and anchor points for each previous task:

This is more accurate but requires memory proportional to the number of past tasks, which becomes problematic in long task sequences. If you are learning a thousand tasks, storing a thousand Fisher vectors and anchor points is memory-intensive. Task-specific EWC is most appropriate when the number of tasks is small and known in advance.

A practical middle ground is to use online EWC with periodic consolidation: every $K$ tasks, merge the accumulated Fisher information and update the anchor point. This trades some approximation accuracy for bounded memory usage while still being more accurate than pure online accumulation.

Synaptic Intelligence (SI), introduced by Zenke et al. (2017), takes a fundamentally different approach to measuring parameter importance. Rather than computing importance from the loss curvature after each task is complete, SI accumulates importance online during training itself. The gradient computation you are already doing in every training step is mined for additional information about which parameters are contributing to learning.

The biological motivation for SI is distinct from EWC's. While EWC is inspired by homeostatic plasticity (stable versus plastic synapses), SI draws from the observation that biological synapses appear to track their own history of activity and use this history to modulate plasticity. Synapses that have recently been active in strengthening memory traces become more resistant to subsequent modification. SI attempts to implement this memory trace computationally.

EWC measures importance by asking: how much does the loss change if we perturb parameter $\theta_i$ at its final value? SI instead asks: how much total contribution did parameter $\theta_i$ make to reducing the loss during the entire training trajectory?

This is a fundamentally different question. EWC cares about the geometry of the loss field at the final point. SI cares about the dynamics of the optimization process. A parameter could be in a flat region of the loss at convergence (low EWC importance) but have played a central role in the optimization path that led there (high SI importance). Conversely, a parameter might be in a high-curvature region at convergence but barely moved during training (high EWC, low SI). Both are valid notions of "importance," but they capture different aspects of the parameter's role in the learned task.

Formally, SI estimates importance as the integral of the product of the gradient and the parameter change along the optimization path. During training on task $T_k$, for each parameter $\theta_i$, SI tracks:

where:

• $\dot{\theta}_i = d\theta_i/dt$: the rate of change of parameter $\theta_i$ during optimization
• $\frac{\partial \mathcal{L}_{T_k}}{\partial \theta_i}$: the gradient of the task loss with respect to $\theta_i$
• The negative sign makes the integral positive when gradients and updates point in the same direction (i.e., when the update reduces the loss)
• The integral is over the entire optimization trajectory from initialization to convergence for task $T_k$

A parameter with high $\omega_i^{(k)}$ was responsible for large loss reductions during training on task $T_k$, which makes it important to preserve. The intuition is that this parameter was "doing work" during optimization: it was moving in a direction aligned with the gradient, and that movement was consequential for reducing the loss.

In practice, since we optimize with discrete gradient steps rather than a continuous trajectory, $\omega_i^{(k)}$ is approximated by summing over training steps:

where:

• $g_i^{(s)}$: the gradient of parameter $\theta_i$ at training step $s$
• $\Delta\theta_i^{(s)} = \theta_i^{(s+1)} - \theta_i^{(s)}$: the change in parameter $\theta_i$ at step $s$
• $S$: the total number of training steps for task $T_k$

The name "path integral" comes from the fact that you are integrating a quantity along the optimization path through parameter space. This is analogous to computing the work done by a force along a trajectory in physics. Just as a force that consistently pushes in the direction of movement does more work than one that pushes sideways, a gradient that consistently points in the direction of parameter updates contributes more to the path integral and thus yields higher importance.

Notice that the path integral depends on the specific trajectory taken by the optimizer, not just on the start and end points. This affects how SI estimates should be interpreted. Two different optimizers (say, SGD and Adam) that converge to the same final parameters will generally produce different SI importance estimates because they take different paths through parameter space. Adam's adaptive learning rates alter the effective direction of each step, changing which gradients align most strongly with which parameter updates. This optimizer-dependence is both a strength and a weakness: it makes SI more sensitive to the actual optimization dynamics, but it also makes the importance estimates harder to interpret theoretically.

After training on task $T_k$, SI computes the final importance measure for parameter $\theta_i$ by normalizing the accumulated gradient-update product:

where:

• $\omega_i^{(k)}$: the accumulated gradient-update product for task $T_k$
• $\Delta_i^{(k)} = \theta_i^{*,k} - \theta_i^{*,k-1}$: the total parameter change from the start to end of task $T_k$
• $\epsilon$: a small constant (e.g., 0.1) added for numerical stability to prevent division by zero
• The denominator normalizes by the squared total displacement, making the measure scale-invariant

This normalization is important for the right reasons. A parameter that changed a lot but also had large gradients throughout should not necessarily be considered more important than one that changed little but always pointed toward the minimum. Dividing by the squared total displacement removes the scale factor of movement, giving you something closer to "per unit of displacement, how much gradient work was done?" The ratio captures the efficiency of each parameter's contribution to learning instead of only its raw magnitude.

The $\epsilon$ in the denominator serves a secondary purpose beyond numerical stability. It provides a floor on the denominator, which prevents parameters that moved very little (near-zero $\Delta_i^{(k)}$) from receiving artificially large importance values. A parameter that was frozen by the optimizer should not be marked as maximally important just because it has a small denominator. The value of $\epsilon$ acts as a threshold: parameters whose total movement is much smaller than $\sqrt{\epsilon}$ get little credit even if their gradient-update products were large.

For continual learning across multiple tasks, SI accumulates importance by summing across all previous tasks:

The total importance grows with each new task that relied on parameter $\theta_i$. This has a natural interpretation: a parameter that has been important for many past tasks should be heavily protected when learning future ones. Unlike task-specific EWC, this accumulated importance does not require storing separate importance vectors per task. You maintain a single vector that grows more refined as more tasks are processed, because importance from earlier tasks is already baked into the accumulation.

With importance weights computed, the SI regularization loss takes the same weighted quadratic form as EWC:

where:

• $c$: the regularization strength hyperparameter (analogous to $\lambda$ in EWC)
• $\theta^*_i$: the parameter value at the start of the current task (i.e., the endpoint of the previous task)
• $\Omega_i$: the accumulated importance from all previous tasks

The practical advantage of SI is that $\omega_i^{(k)}$ is computed incrementally during training, requiring just two additional scalar quantities per parameter: the running sum and the current parameter value at task start. No separate backward pass over task data is needed after training, unlike EWC which requires re-running the training data to compute the Fisher. For scenarios where the training data cannot be stored or accessed again after training (which is common in privacy-sensitive applications), SI's online accumulation is a significant practical advantage.

The downside is implementation complexity. You must carefully instrument the training loop to accumulate gradient-update products at every step, and you must correctly reset the accumulators at task boundaries. Any bug in this accounting will silently produce incorrect importance estimates, leading to more or less forgetting than expected. EWC's importance computation is a clean, separate pass that is easier to validate in isolation.

EWC and SI measure parameter importance from different vantage points. Understanding when each is more appropriate requires grasping what their importance estimates capture, and what each is blind to.

EWC measures local curvature at convergence. The Fisher information captures how the loss surface curves around the final parameter values. High curvature in a direction means the loss is sensitive to changes there. This is a static measurement taken after task training is complete. EWC's importance estimate says nothing about how the optimization got to the final point; it only cares about the geometry at that point. This makes EWC well-suited for cases where the loss landscape is well-behaved and the final parameter configuration is informative about sensitivity.

SI measures trajectory-integrated contribution. The gradient-update integral captures how much each parameter participated in the descent during training. A parameter that consistently pointed in the direction of improvement while also moving substantially gets high importance. This is a dynamic measurement accumulated throughout training. SI's importance estimate carries history: it knows which parameters were active drivers of learning throughout the optimization, not just what the local geometry looks like at the end.

These perspectives can disagree. A parameter might have high curvature at the final point (high EWC importance) but have barely moved during training (low SI importance), or vice versa. In practice, both methods tend to identify similar parameters as important for typical deep learning tasks, but their behavior can diverge in specific scenarios. For example, in tasks with flat loss landscapes near the optimum (which can occur with overparameterized models), EWC would assign low importance to many parameters even if they were critical during training, while SI would correctly identify them as important. Conversely, in tasks where the optimizer made many exploratory moves before settling on a solution, SI might accumulate importance on parameters that were active early in training but became less relevant by the end.

One important practical difference concerns computational cost and timing. EWC requires a full pass over the task data to compute the Fisher after each task, which costs roughly the same as one training epoch. SI requires only constant-time operations per gradient step, making it cheaper per-step, but requires careful implementation to accumulate $\omega_i$ correctly throughout training. If the training data for previous tasks is not retained (which is the typical assumption in continual learning settings where privacy constraints prohibit keeping past data), EWC's post-hoc Fisher computation is impossible; SI does not have this limitation.

Another difference is sensitivity to the optimization trajectory. EWC's importance is independent of the path taken during optimization; it only cares about the final point. SI's importance depends entirely on the path: two different optimizers that converge to the same parameters but take different routes will produce different SI importance estimates. This makes SI's behavior harder to predict and analyze theoretically, but also potentially more informative in capturing which parameters drove the learning.

From a memory perspective, both methods require $O(|\theta|)$ storage for the importance vector and reference parameters, putting them in the same class. The practical advantage of SI is that it maintains a single accumulated importance vector regardless of the number of tasks, whereas task-specific EWC's memory grows linearly with the number of tasks. Online EWC matches SI's memory efficiency but with the approximation errors discussed earlier.

To make the regularization intuition concrete, consider a small network trained sequentially on two classification tasks. After training on task A, we compute importance weights. When training on task B, parameters with high importance from task A are penalized for deviating from their task-A values.

Consider a single parameter $\theta_j$ with:

• Task A optimum: $\theta^*_{A,j} = 0.8$
• EWC importance: $F_{jj} = 5.0$ (high: this parameter matters a lot for task A)
• Current value during task B training: $\theta_j = 0.5$

The EWC regularization penalty contribution from this single parameter is:

This penalty gradient pushes $\theta_j$ back toward 0.8. The gradient of the regularization term with respect to $\theta_j$ is:

This negative gradient pushes $\theta_j$ upward, toward its original value of 0.8. The total gradient for this parameter during task B training is the sum of the task B gradient and this restoring force. If task B's gradient for $\theta_j$ is small (say, $-0.1$), the EWC restoring force dominates and the parameter stays near its task A optimum. If task B strongly needs to change $\theta_j$ (say, its gradient is $-3.0$), task B's learning will partially override the protection. The interplay between the task gradient and the restoring force determines the final parameter value after each step, and the network will converge to a compromise point where neither force is fully satisfied.

Now consider a second parameter $\theta_k$ with importance $F_{kk} = 0.05$ (low: this parameter barely affected task A's loss). Even a large displacement, say $\theta_k - \theta^*_{A,k} = 2.0$, produces only a small penalty:

This parameter is essentially free to adapt to task B. The network can repurpose it entirely without incurring significant regularization cost. This is the key mechanism: the model's plasticity is redirected toward parameters that are available (low importance) while rigidity is enforced on parameters that are essential (high importance). Rather than distributing learning uniformly across all weights, EWC carves out a subspace of available parameters and confines the new task's learning to that subspace. The available subspace is exactly the set of parameters with low Fisher values for previous tasks.

The following visualization shows how EWC importance weights shape the effective learning rate across parameters. Parameters with high Fisher values face a strong restoring force that partially cancels the task B gradient; parameters with near-zero Fisher values can update freely.

Parameters with Fisher value 5.0 experience a restoring gradient five times larger than those with Fisher value 1.0 for any given displacement. This multiplicative scaling is what differentiates EWC from uniform $L_2$ regularization, which would apply the same slope for all parameters. Notice also that the restoring gradient grows linearly with displacement: the further you push a protected parameter from its anchor, the harder it pushes back. This Hookean behavior is exactly the "elastic" quality that gives EWC its name.

Let's implement both EWC and SI from scratch using PyTorch and evaluate their ability to prevent catastrophic forgetting on a two-task sequential learning scenario.

We begin by importing the necessary libraries and creating a simple two-task dataset. We will use a multi-layer perceptron trained first on a binary classification task, then adapted to a second task.

Task A classifies points based on a linear boundary (sign of $x_1 - x_2$), while task B uses a radial boundary (distance from the origin). These two tasks require different decision surfaces, so the network must learn different computations for each, making them a good stress test for catastrophic forgetting protection. The linear boundary for task A can be solved by a single linear layer; the radial boundary for task B requires nonlinear interactions between the two input features. A network that finds its weights adjusted to solve task A will generally need to reconfigure its internal representations to solve task B, creating conditions for forgetting.

We define a simple two-layer MLP. Keeping the architecture small makes the catastrophic forgetting effect visible and the importance weights interpretable.

The MLP has two hidden layers with 64 units each, separated by ReLU activations. The output is a single logit, and we use binary cross-entropy with logits as the loss function. This architecture is deliberately simple so that the importance weight distributions and forgetting behavior are easy to observe without being obscured by the complexities of larger networks.

First, let's see the extent of catastrophic forgetting when we train sequentially without any protection.

The drop in task A accuracy after training on task B is catastrophic forgetting in action. The model learned task B effectively but lost much of its task A knowledge in the process. This is not a subtle degradation; the drop is substantial, corresponding to a model that has largely forgotten the linear boundary it learned. From the optimizer's perspective, this is entirely rational: it minimized the loss on task B as effectively as possible. It simply had no incentive to preserve task A.

Now let's implement EWC. The key component is computing the diagonal Fisher information matrix after training on task A.

In this particular run, EWC leaves task A forgetting close to the unprotected baseline while still achieving meaningful task B accuracy. The Fisher-weighted penalty changes which parameters the optimizer can move, but the chosen regularization strength does not produce a large retention gain here. This result reflects the method's sensitivity to $\lambda$: some degree of compromise is inevitable when a single parameter vector must serve two different tasks simultaneously.

After computing the diagonal Fisher, we can visualize which parameters the model considers most important for task A. High-importance parameters will be strongly protected during task B training.

The distribution reveals an important property: regularization-based methods work because importance is concentrated in a subset of parameters. The isolated bar at $-12$ contains exact zeros after the numerical floor used for the logarithm; those parameters receive essentially no EWC protection. The continuous mass to its right spans several orders of magnitude, with the right tail representing parameters that most strongly encode task A's solution. If importance were uniformly high, every parameter would require protection and the model could not adapt to new tasks at all. This concentration reflects the fact that neural networks are typically over-parameterized relative to the tasks they solve, leaving many parameters available for repurposing.

Synaptic Intelligence accumulates importance during training rather than computing it post-hoc. This requires modifying the training loop to track per-parameter gradient-update products.

SI also reduces forgetting compared to the baseline. The performance characteristics differ from EWC because the importance estimates reflect the optimization trajectory rather than the final loss curvature. Whether SI or EWC performs better on any particular problem depends on the tasks' structure, the optimizer choice, and the hyperparameter settings.

Let's collect the results and compare all three approaches.

The bar charts illustrate the stability-plasticity trade-off in concrete terms. The baseline shows a large drop from the blue bar to the orange bar in the left panel, indicating severe forgetting. EWC's gap is nearly the same at this setting, whereas SI produces a visibly smaller drop. The right panel quantifies that difference: EWC improves the baseline by only 0.005 accuracy points, while SI improves it by 0.095.

The regularization strength hyperparameter controls the stability-plasticity balance directly. The following plot shows how task A retention and task B performance change as we vary $\lambda$ for EWC, using a sweep over different values.

In this sweep, stronger regularization eventually improves task A retention, but the gain is modest until the largest $\lambda$. Task B accuracy is comparatively flat through the middle of the sweep and then drops as the constraint becomes dominant. The curves do not cross, so there is no single visually defined optimum: choosing $\lambda$ requires an explicit objective, such as a weighted average of both accuracies or a minimum acceptable score for each task. Similar tasks can tolerate stronger regularization because they benefit from more compatible parameter configurations, while dissimilar tasks may need a lower $\lambda$ to preserve enough plasticity for task B.

The key hyperparameters for both methods are:

• lambda_ewc / c_si: The regularization strength. Higher values preserve more of task A but allow less adaptation to task B. This hyperparameter requires careful tuning; too high and the model cannot learn new tasks, too low and forgetting is not prevented.
• epochs: Training epochs per task. Both methods benefit from sufficient training to reach convergence before computing importance or transitioning to the next task.
• xi (SI only): The numerical stability constant in the SI importance normalization. Small values (0.01 to 0.1) work well in practice.
• learning rate: SI with SGD typically needs a different learning rate than EWC with Adam, since the optimizer choice affects the path integral accumulation in SI.

Regularization-based continual learning methods occupy a specific point in the design space defined by two opposing forces: the stability-plasticity dilemma. Stability refers to the ability to retain previously learned knowledge; plasticity refers to the ability to learn new tasks. Any regularization method that makes weights harder to change increases stability at the cost of plasticity. This is not a failure of implementation; it is a logical consequence of using a single parameter vector for all tasks. The vector can only be in one place at a time, and the more tightly you anchor it to past task optima, the less freedom you leave for new tasks.

The regularization strength hyperparameter ($\lambda$ in EWC, $c$ in SI) directly controls this trade-off. Setting it too low gives a solution close to vanilla sequential training with significant forgetting. Setting it too high produces a model that can barely learn new tasks because every useful parameter is locked in place. In practice, the optimal value depends on how similar the tasks are, how many tasks are in the sequence, and how large the model is relative to the tasks' complexity. Large models have more redundant parameters and thus more free capacity; the optimal regularization strength for a large model is typically higher than for a small model trained on the same task sequence, because there are more parameters available for new tasks even after the important ones are protected.

Regularization methods share a common structural limitation: they assume that the optimal parameters for all tasks can be represented in the same weight space simultaneously. For tasks that require fundamentally different computations, this assumption may fail. If task A needs high values in certain weights and task B needs low values in the same weights, no regularization scheme can satisfy both constraints without a performance penalty on one or both tasks. The loss landscape overlap problem is not always visible from the Fisher values or SI importance scores; you may not know that tasks are structurally incompatible until you observe the actual performance on both tasks after training. In such cases, architecture methods (which we will explore in the next chapter) offer a different solution: allocating separate parameter subsets to different tasks rather than trying to find a single parameter configuration that serves all tasks.

Another limitation concerns scalability to long task sequences. As the number of tasks grows, accumulated importance weights from many previous tasks can make nearly every parameter appear important, effectively freezing the model. If every parameter has been critical for at least one past task, the importance vector has no zeros and the regularization prevents any learning at all. Online EWC mitigates this by merging Fisher matrices, but at the cost of approximation accuracy. SI naturally handles this through its normalization, but can suffer from gradient signal dilution in very long task sequences where early tasks' contributions become small relative to the accumulated sum. Replay methods, which we will cover after architecture methods, sidestep this problem entirely by maintaining explicit memory of past data, allowing the model to remain fully plastic with respect to current task objectives while the replay mechanism ensures past tasks are not forgotten.

Regularization methods also tend to assume clear task boundaries. EWC can only compute Fisher information at the end of a clearly demarcated task, and SI requires knowing when to finalize the importance update. In settings where task boundaries are unknown (referred to as task-agnostic continual learning), these methods require additional heuristics to determine when to consolidate importance weights. Some researchers have proposed online Fisher estimation that gradually updates the importance measure throughout training, but this introduces new questions about how to weight recent versus older gradients.

A further practical concern is sensitivity to the Fisher approximation quality. The diagonal Fisher assumes that all parameters are independent, which is generally false. In practice, parameters in the same layer have correlated gradients, especially in the final classification layer where the output neurons must maintain specific relationships with each other. The diagonal approximation ignores these correlations, potentially over-protecting some parameters and under-protecting others. More accurate Kronecker-factored approximations (such as K-FAC) exist but significantly increase computational and memory overhead.

Despite these limitations, EWC and SI represent an important class of continual learning solutions. They require no additional storage of past data, have reasonable computational overhead, and can substantially reduce catastrophic forgetting in many practical settings. They are also theoretically principled, which makes them valuable baselines and building blocks for more advanced approaches. Any researcher working on continual learning should understand these methods thoroughly, because they define the problem space and set a clear benchmark against which more complex approaches must be compared.

Regularization methods prevent catastrophic forgetting by adding a penalty term to the loss that discourages large changes to parameters important for past tasks. The key concepts from this chapter are:

• Weighted regularization: All regularization-based methods use the form $\sum_i \Omega_i (\theta_i - \theta^*_i)^2$, where $\Omega_i$ measures parameter importance and $\theta^*_i$ is the reference value from previous tasks. Uniform $L_2$ regularization is the degenerate case where all $\Omega_i$ are equal; it protects everything and therefore prevents new learning.

• EWC uses the diagonal Fisher information to measure importance. The Fisher captures the curvature of the loss surface around the previous optimum, telling you how sensitive task performance is to changes in each parameter. It is derived from a Laplace approximation to the Bayesian posterior and computed with one backward pass over the previous task's data after training. The diagonal approximation is the key practical concession that makes the method scalable.

• SI uses path integrals to accumulate importance online during training. The gradient-update product at each step measures each parameter's contribution to reducing the loss, and these contributions are summed over the entire optimization trajectory. SI requires no post-hoc data access but needs careful implementation to accumulate integrals correctly. Its importance estimates depend on the optimizer used, making them more dynamic but less theoretically transparent than EWC's Fisher-based weights.

• The stability-plasticity trade-off governs all regularization approaches. The regularization strength hyperparameter controls whether the model prioritizes remembering old tasks or learning new ones. There is no free lunch: you cannot simultaneously maximize both. The optimal setting depends on task similarity, model capacity, and the number of tasks in the sequence.

• Structural limitations: Regularization methods work well when tasks are compatible (their optima lie close together in parameter space), but struggle when tasks require conflicting parameter configurations or when the number of tasks grows very large. As importance weights accumulate across many tasks, the effective plasticity of the model decreases, potentially preventing useful learning on future tasks.

• Practical considerations: EWC requires access to previous task data for Fisher computation, while SI does not. Both methods store importance weights and reference parameters proportional to the number of network parameters. EWC can extend to multiple tasks via online Fisher accumulation with constant memory cost, though with approximation trade-offs.

In the next chapter on Replay Methods, we will see how storing and replaying samples from previous tasks offers a complementary approach to catastrophic forgetting, one that maintains explicit episodic memory rather than relying on importance-weighted penalties. Replay sidesteps the structural limitations of regularization by ensuring the model always has direct gradient signal from past tasks, but introduces its own challenges around storage efficiency and privacy.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about regularization methods for continual learning.