Regularization Methods › Regularization Trade-offs and Limitations

Regularization-based continual learning methods occupy a specific point in the design space defined by two opposing forces: the stability-plasticity dilemma. Stability refers to the ability to retain previously learned knowledge; plasticity refers to the ability to learn new tasks. Any regularization method that makes weights harder to change increases stability at the cost of plasticity. This is not a failure of implementation; it is a logical consequence of using a single parameter vector for all tasks. The vector can only be in one place at a time, and the more tightly you anchor it to past task optima, the less freedom you leave for new tasks.

The regularization strength hyperparameter ($\lambda$ in EWC, $c$ in SI) directly controls this trade-off. Setting it too low gives a solution close to vanilla sequential training with significant forgetting. Setting it too high produces a model that can barely learn new tasks because every useful parameter is locked in place. In practice, the optimal value depends on how similar the tasks are, how many tasks are in the sequence, and how large the model is relative to the tasks' complexity. Large models have more redundant parameters and thus more free capacity; the optimal regularization strength for a large model is typically higher than for a small model trained on the same task sequence, because there are more parameters available for new tasks even after the important ones are protected.

Regularization methods share a common structural limitation: they assume that the optimal parameters for all tasks can be represented in the same weight space simultaneously. For tasks that require fundamentally different computations, this assumption may fail. If task A needs high values in certain weights and task B needs low values in the same weights, no regularization scheme can satisfy both constraints without a performance penalty on one or both tasks. The loss landscape overlap problem is not always visible from the Fisher values or SI importance scores; you may not know that tasks are structurally incompatible until you observe the actual performance on both tasks after training. In such cases, architecture methods (which we will explore in the next chapter) offer a different solution: allocating separate parameter subsets to different tasks rather than trying to find a single parameter configuration that serves all tasks.

Another limitation concerns scalability to long task sequences. As the number of tasks grows, accumulated importance weights from many previous tasks can make nearly every parameter appear important, effectively freezing the model. If every parameter has been critical for at least one past task, the importance vector has no zeros and the regularization prevents any learning at all. Online EWC mitigates this by merging Fisher matrices, but at the cost of approximation accuracy. SI naturally handles this through its normalization, but can suffer from gradient signal dilution in very long task sequences where early tasks' contributions become small relative to the accumulated sum. Replay methods, which we will cover after architecture methods, sidestep this problem entirely by maintaining explicit memory of past data, allowing the model to remain fully plastic with respect to current task objectives while the replay mechanism ensures past tasks are not forgotten.

Regularization methods also tend to assume clear task boundaries. EWC can only compute Fisher information at the end of a clearly demarcated task, and SI requires knowing when to finalize the importance update. In settings where task boundaries are unknown (referred to as task-agnostic continual learning), these methods require additional heuristics to determine when to consolidate importance weights. Some researchers have proposed online Fisher estimation that gradually updates the importance measure throughout training, but this introduces new questions about how to weight recent versus older gradients.

A further practical concern is sensitivity to the Fisher approximation quality. The diagonal Fisher assumes that all parameters are independent, which is generally false. In practice, parameters in the same layer have correlated gradients, especially in the final classification layer where the output neurons must maintain specific relationships with each other. The diagonal approximation ignores these correlations, potentially over-protecting some parameters and under-protecting others. More accurate Kronecker-factored approximations (such as K-FAC) exist but significantly increase computational and memory overhead.

Despite these limitations, EWC and SI represent an important class of continual learning solutions. They require no additional storage of past data, have reasonable computational overhead, and can substantially reduce catastrophic forgetting in many practical settings. They are also theoretically principled, which makes them valuable baselines and building blocks for more advanced approaches. Any researcher working on continual learning should understand these methods thoroughly, because they define the problem space and set a clear benchmark against which more complex approaches must be compared.