Regularization Methods › EWC vs SI: A Conceptual Comparison

EWC and SI measure parameter importance from different vantage points. Understanding when each is more appropriate requires grasping what their importance estimates capture, and what each is blind to.

EWC measures local curvature at convergence. The Fisher information captures how the loss surface curves around the final parameter values. High curvature in a direction means the loss is sensitive to changes there. This is a static measurement taken after task training is complete. EWC's importance estimate says nothing about how the optimization got to the final point; it only cares about the geometry at that point. This makes EWC well-suited for cases where the loss landscape is well-behaved and the final parameter configuration is informative about sensitivity.

SI measures trajectory-integrated contribution. The gradient-update integral captures how much each parameter participated in the descent during training. A parameter that consistently pointed in the direction of improvement while also moving substantially gets high importance. This is a dynamic measurement accumulated throughout training. SI's importance estimate carries history: it knows which parameters were active drivers of learning throughout the optimization, not just what the local geometry looks like at the end.

These perspectives can disagree. A parameter might have high curvature at the final point (high EWC importance) but have barely moved during training (low SI importance), or vice versa. In practice, both methods tend to identify similar parameters as important for typical deep learning tasks, but their behavior can diverge in specific scenarios. For example, in tasks with flat loss landscapes near the optimum (which can occur with overparameterized models), EWC would assign low importance to many parameters even if they were critical during training, while SI would correctly identify them as important. Conversely, in tasks where the optimizer made many exploratory moves before settling on a solution, SI might accumulate importance on parameters that were active early in training but became less relevant by the end.

One important practical difference concerns computational cost and timing. EWC requires a full pass over the task data to compute the Fisher after each task, which costs roughly the same as one training epoch. SI requires only constant-time operations per gradient step, making it cheaper per-step, but requires careful implementation to accumulate $\omega_i$ correctly throughout training. If the training data for previous tasks is not retained (which is the typical assumption in continual learning settings where privacy constraints prohibit keeping past data), EWC's post-hoc Fisher computation is impossible; SI does not have this limitation.

Another difference is sensitivity to the optimization trajectory. EWC's importance is independent of the path taken during optimization; it only cares about the final point. SI's importance depends entirely on the path: two different optimizers that converge to the same parameters but take different routes will produce different SI importance estimates. This makes SI's behavior harder to predict and analyze theoretically, but also potentially more informative in capturing which parameters drove the learning.

From a memory perspective, both methods require $O(|\theta|)$ storage for the importance vector and reference parameters, putting them in the same class. The practical advantage of SI is that it maintains a single accumulated importance vector regardless of the number of tasks, whereas task-specific EWC's memory grows linearly with the number of tasks. Online EWC matches SI's memory efficiency but with the approximation errors discussed earlier.