Online, Offline, and Continual Learning › Catastrophic Forgetting

Catastrophic forgetting is the abrupt loss of previously learned behavior when a neural network is trained on new data. It was documented in connectionist models in the late 1980s (McCloskey and Cohen, 1989; Ratcliff, 1990) and has remained a central obstacle ever since.

The phenomenon is easy to state. Train a network on task A until it performs well. Then train the same network on task B. Performance on task A frequently collapses to near chance, even though the network has more than enough capacity to represent both tasks. The parameters that encoded A were overwritten by the gradients that encode B.

For a world model the consequences are concrete. A model that has learned the dynamics of a pick-and-place task may lose them entirely after being fine-tuned on a new task with a different gripper. A model that has learned how objects fall may lose that knowledge after adapting to a scene where nothing falls. The loss is not graceful degradation; it is erasure.

The word "catastrophic" is chosen deliberately. In many optimization problems, adding new data gently shifts the solution while preserving most of its useful structure. In neural networks trained by gradient descent, adding a new task can shift the solution so sharply that old behavior disappears. The network still has the capacity to represent the old task. The problem is that the parameters have moved to a region of parameter space that represents the new task well and the old task poorly.

Gradient descent has no notion of importance. Every parameter is adjusted in whatever direction reduces the current loss, and a parameter that was critical for task A is treated no differently from one that was never used. When tasks require overlapping parameters but conflicting values, the update for B necessarily damages A.

There is a geometric way to see this. The gradient for task B is a direction in parameter space that decreases $L_B$. Whether it also decreases $L_A$ depends on the angle between the two gradients. When tasks are related, the angle is small and interference is mild. When tasks conflict, the angle is obtuse and progress on one is regression on the other. Catastrophic forgetting is severe gradient interference.

The geometry also explains why forgetting is not always total. If the gradients for two tasks are nearly orthogonal, the update for one has little effect on the other, and the network can learn both without interference. If the gradients are aligned, learning one task may help the other, which is positive transfer. The hard case is when the gradients point in opposite directions on the same parameters. Then any step toward the new task is a step away from the old one.

The underlying tension was named by Grossberg in the 1980s. A learning system must be stable enough to retain old knowledge and plastic enough to acquire new knowledge. These requirements pull in opposite directions.

• Perfect stability means the model never changes, which means it cannot learn anything new.
• Perfect plasticity means the model always changes, which means it retains nothing.

Every continual learning method is a particular point on this spectrum. The engineering question is not whether to be stable or plastic, but where on the spectrum to sit given how fast the world is expected to change and how costly forgetting is expected to be.

The dilemma is not unique to neural networks. Any learning system that updates its parameters in response to new data faces a version of it. What makes neural networks special is the density of their representations: many tasks share parameters, and a single parameter can be important for many behaviors. That sharing is what gives neural networks their generalization power, and it is also what makes them vulnerable to interference.

Methods for mitigating forgetting fall into three broad families. In practice they are combined.

Regularization-based methods penalize changes to parameters that were important for previous tasks. The canonical example is Elastic Weight Consolidation (Kirkpatrick et al., 2017). After training on task A, EWC estimates how sensitive the loss is to each parameter using the diagonal of the Fisher information matrix $F$, and adds a quadratic penalty when training on task B:

$$L(\theta) = L_B(\theta) + \frac{\lambda_{\mathrm{EWC}}}{2}\sum_i F_i\,\big(\theta_i - \theta_{A,i}^{\star}\big)^2,$$

where:

• $L(\theta)$: the total loss while training on task B, combining the task loss with a quadratic penalty
• $L_B(\theta)$: the loss on task B alone
• $\lambda_{\mathrm{EWC}}$: the regularization strength, controlling how strongly the penalty anchors important parameters
• $F_i$: the Fisher information for parameter $i$, measuring how important that parameter was for task A
• $\theta_i$: the current value of parameter $i$
• $\theta_{A,i}^{\star}$: the value of parameter $i$ at the end of task A
• $\theta$: the parameter vector
• $i$: the parameter index

The Fisher value $F_i$ itself is defined as

$$F_i = \mathbb{E}_{(x, y) \sim \mathcal{D}_A}\!\left[\left(\frac{\partial}{\partial \theta_i}\log p_\theta(y \mid x)\right)^{\!2}\right]_{\theta = \theta_A^{\star}},$$

where:

• $(x, y) \sim \mathcal{D}_A$: input-target pairs drawn from the task-A dataset
• $x$: an input (transition) drawn from task A
• $y$: the target (next state) paired with $x$
• $p_\theta(y \mid x)$: the model's predictive distribution over targets given inputs
• $\partial / \partial \theta_i$: the partial derivative with respect to parameter $i$
• $\theta = \theta_A^{\star}$: the parameters frozen at the end of task A
• $\theta$: the model parameters
• $\theta_i$: the $i$-th parameter
• $\mathcal{D}_A$: the task-A dataset

The Fisher value $F_i$ is large for parameters the model relies on and small for parameters it does not. The penalty is therefore a soft, anisotropic spring anchoring important parameters to their old values while leaving unimportant ones free to move. Related methods include Synaptic Intelligence (Zenke et al., 2017) and Memory Aware Synapses (Aljundi et al., 2018), which differ mainly in how they estimate importance.

The key intuition behind EWC is that not all parameters are equally important for a task. Near a fitted probabilistic model, and under the approximations used to derive EWC, the Fisher supplies a local curvature-based importance measure. Penalizing movement in high-Fisher directions can preserve old-task performance while leaving lower-Fisher directions more flexible; it does not guarantee that every direction behaves this way away from the anchor.

Replay-based methods keep a buffer of old examples, or a generative model of them, and mix them into the new task's batches. Gradient Episodic Memory (Lopez-Paz and Ranzato, 2017) goes further by constraining the update using episodic-memory losses from previous tasks. Averaged GEM (Chaudhry et al., 2019) replaces the per-task constraints with one constraint based on an averaged reference gradient, reducing computation and memory.

The projection in GEM is a geometric operation. It takes the current-task gradient and, when necessary, projects it so its first-order inner product with each previous task's episodic-memory gradient is nonnegative. This constrains first-order increases in the aggregate memory loss for each previous task; it is not a per-example or finite-step guarantee. The projected gradient may be smaller than the raw gradient, slowing learning of the new task.

Architectural methods isolate parameters used by different tasks. Progressive Networks (Rusu et al., 2016) freeze old columns and add a new column with lateral connections, so capacity grows while earlier features remain available for transfer. PackNet (Mallya and Lazebnik, 2018) instead iteratively prunes and freezes weights within a fixed network, using task-specific masks and the remaining free capacity. Both protect allocated weights, but their capacity and routing costs differ.

The architectural approach trades capacity and routing complexity for reduced interference. Progressive Networks explicitly use frozen old features through lateral connections and grow with the number of tasks. PackNet keeps model size fixed but consumes free parameters and requires the correct task mask at inference. Protecting parameters prevents direct overwriting of those parameters, though end-to-end retention still depends on routing and evaluation conditions.

To compare methods you need metrics. Let $R_{i,j}$ denote performance on task $j$ after training through task $i$, with $T$ tasks total. Three standard measures are:

$$\begin{aligned} \text{ACC} &= \frac{1}{T}\sum_{j=1}^{T} R_{T,j}, \\ \text{BWT} &= \frac{1}{T-1}\sum_{j=1}^{T-1}\big(R_{T,j} - R_{j,j}\big), \\ \text{FWT} &= \frac{1}{T-1}\sum_{j=2}^{T}\big(R_{j-1,j} - b_j\big). \end{aligned}$$

where:

• $R_{i,j}$: performance on task $j$ after training through task $i$ (so $R_{j,j}$ is performance on task $j$ right after learning it, and $R_{T,j}$ is its performance after all $T$ tasks)

• $T$: the total number of tasks

• $\text{ACC}$: average accuracy across all tasks after training through task $T$

• $\text{BWT}$: backward transfer, the average change in performance on earlier tasks after learning later ones

• $\text{FWT}$: forward transfer, performance on a new task before training it relative to a specified baseline

• $b_j$: baseline performance on task $j$; in the original GEM definition, test performance at random initialization

• $i$: the task index up to which training has proceeded

• $j$: the task index being evaluated

• ACC is final average performance across all tasks. Higher is better.

• BWT, backward transfer, measures how much learning later tasks changed performance on earlier ones. Negative BWT is forgetting. Positive BWT means the later tasks helped.

• FWT, forward transfer, measures how much earlier tasks change performance on a new task before it is trained, relative to $b_j$. In the original GEM metric, $b_j$ is random-initialization performance. Positive FWT indicates beneficial transfer under that baseline.

Note that these metrics are computed on a held-out evaluation set per task, not on the training data. A method that reduces training loss on task A while degrading held-out performance has not avoided forgetting; it has overfit.

The three metrics capture different aspects of continual learning. ACC measures final performance, which is what ultimately matters for deployment. BWT measures whether the model retained old skills. FWT measures whether the model can use old skills to learn new ones faster. A method with high ACC might have poor BWT if the final task is easy and the earlier tasks are irrelevant. A method with high BWT might have low ACC if it is too conservative to learn the new task well. The metrics must be read together, and they must be computed on data that the model has not trained on.

A subtlety that the continual learning literature often understates: forgetting is not always a bug. If the world changed, the old dynamics are no longer true, and a model that insists on them is wrong. The distinction is between forgetting knowledge that is still valid and forgetting knowledge that has become obsolete.

The practical difficulty is that the agent usually cannot tell which is which. A drop in performance on old data could mean the model degraded, or it could mean the old data is no longer representative. Resolving this ambiguity is one of the central problems of continual adaptation, and it is the subject of the next section.

This ambiguity has an important consequence for method design. A method that aggressively prevents forgetting may be preserving obsolete knowledge. A method that aggressively allows forgetting may be discarding valid knowledge. The right balance depends on the environment's rate of change and on the agent's ability to detect change. In practice, the best methods often combine a mechanism for retaining knowledge with a mechanism for detecting when retention is no longer appropriate.