Regularization Methods › Summary

Regularization methods prevent catastrophic forgetting by adding a penalty term to the loss that discourages large changes to parameters important for past tasks. The key concepts from this chapter are:

• Weighted regularization: All regularization-based methods use the form $\sum_i \Omega_i (\theta_i - \theta^*_i)^2$, where $\Omega_i$ measures parameter importance and $\theta^*_i$ is the reference value from previous tasks. Uniform $L_2$ regularization is the degenerate case where all $\Omega_i$ are equal; it protects everything and therefore prevents new learning.

• EWC uses the diagonal Fisher information to measure importance. The Fisher captures the curvature of the loss surface around the previous optimum, telling you how sensitive task performance is to changes in each parameter. It is derived from a Laplace approximation to the Bayesian posterior and computed with one backward pass over the previous task's data after training. The diagonal approximation is the key practical concession that makes the method scalable.

• SI uses path integrals to accumulate importance online during training. The gradient-update product at each step measures each parameter's contribution to reducing the loss, and these contributions are summed over the entire optimization trajectory. SI requires no post-hoc data access but needs careful implementation to accumulate integrals correctly. Its importance estimates depend on the optimizer used, making them more dynamic but less theoretically transparent than EWC's Fisher-based weights.

• The stability-plasticity trade-off governs all regularization approaches. The regularization strength hyperparameter controls whether the model prioritizes remembering old tasks or learning new ones. There is no free lunch: you cannot simultaneously maximize both. The optimal setting depends on task similarity, model capacity, and the number of tasks in the sequence.

• Structural limitations: Regularization methods work well when tasks are compatible (their optima lie close together in parameter space), but struggle when tasks require conflicting parameter configurations or when the number of tasks grows very large. As importance weights accumulate across many tasks, the effective plasticity of the model decreases, potentially preventing useful learning on future tasks.

• Practical considerations: EWC requires access to previous task data for Fisher computation, while SI does not. Both methods store importance weights and reference parameters proportional to the number of network parameters. EWC can extend to multiple tasks via online Fisher accumulation with constant memory cost, though with approximation trade-offs.

In the next chapter on Replay Methods, we will see how storing and replaying samples from previous tasks offers a complementary approach to catastrophic forgetting, one that maintains explicit episodic memory rather than relying on importance-weighted penalties. Replay sidesteps the structural limitations of regularization by ensuring the model always has direct gradient signal from past tasks, but introduces its own challenges around storage efficiency and privacy.