Continual Learning Problem › Catastrophic Forgetting

The phenomenon of catastrophic forgetting (also called catastrophic interference) was first described by McCloskey and Cohen in 1989, and later studied extensively by Ratcliff (1990) and French (1999). When a neural network learns new information, the gradient updates that encode that new information overwrite the weights that encoded previous information. The overwriting is not gradual or selective; it is typically rapid and near-total. This is the "catastrophic" part of the name.

What makes this so striking is the contrast with biological memory. When you learn a new recipe, you do not forget all previous recipes. When you learn a new song, you do not lose the ability to play previously learned songs. Human memory is susceptible to interference, but the degree of interference is nowhere near what artificial neural networks experience. A neural network trained sequentially on two tasks can lose essentially all performance on the first task within dozens of gradient steps on the second, even when the two tasks appear superficially similar to a human observer.

To understand why forgetting happens, consider what gradient descent does. When training on task $T_1$, the optimizer moves the parameters $\theta$ to minimize $\mathcal{L}_1(\theta)$. The loss surface for task 1 has a basin (a region of low loss) that the optimizer settles into. The parameters at this optimum are tuned precisely to the features of task 1's data distribution: the weight magnitudes and directions encode the statistical regularities of that particular task.

When training then shifts to task $T_2$, the optimizer now minimizes $\mathcal{L}_2(\theta)$, which has a completely different loss surface. Gradient descent moves parameters toward the minimum of $\mathcal{L}_2$, regardless of where the minimum of $\mathcal{L}_1$ was. Unless the two loss surfaces happen to share a low-loss region, moving toward the task 2 minimum means moving away from the task 1 minimum. The model is not "deciding" to forget task 1; it is simply following gradients that have no knowledge of or concern for task 1 performance.

This can be visualized in parameter space. Imagine two elliptical basins in a 2D parameter landscape. The optimizer first descends into basin 1. Then it starts descending toward basin 2. If these basins are separated, the model's parameters end up in basin 2, far from where they performed well on task 1. The model has forgotten task 1, not because of any design failure, but because that is exactly what unconstrained gradient descent on the task 2 objective will do.

In high-dimensional parameter spaces (billions of parameters in large language models), the situation is more complex but the same principle applies. Each task carves out a region of parameter space where its performance is high. Sequential gradient updates do not respect the boundaries of previous tasks' performance regions. The optimizer is blind to them, because the loss function it is minimizing does not include any term that penalizes moving outside those regions.

The two basins in @fig-loss-landscape-forgetting do not overlap. That is the crux of the problem. For a simple two-task, two-parameter toy example, you can imagine a joint basin that satisfies both tasks, and indeed in very high-dimensional spaces such overlap does exist. But finding that overlap through gradient descent on only one task's loss is not guaranteed, and in practice the optimizer takes the path of least resistance along the gradient, which is the path toward the current task's minimum and away from everything else.

The forgetting problem is exacerbated by how neural networks distribute their representations. In a fully connected network, every weight participates in encoding information about every input. When you learn task 2, updates to shared weights inevitably disrupt the encoding of task 1's features. There is no structural boundary that separates "task 1 weights" from "task 2 weights"; the same matrix entries that encode syntax for one task also encode vocabulary statistics for another. Gradient descent on task 2 updates all of them simultaneously.

This is fundamentally different from how biological brains handle sequential learning. Neuroscience research suggests that the hippocampus acts as a fast-learning memory buffer that consolidates new experiences rapidly, while the neocortex learns slowly and represents general patterns extracted over many episodes. This complementary learning systems theory, proposed by McClelland, McNaughton, and O'Reilly in 1995, posits that the brain's resistance to catastrophic forgetting comes precisely from this two-speed architecture. New memories are held in a volatile fast store while gradually transferred to a stable slow store, preventing the sudden interference that plagues single-speed learners.

This biological insight directly inspired experience replay methods in continual learning, which maintain a small buffer of past examples and interleave them with new data during training. The replay buffer functions like a simplified hippocampus, giving a fast retrieval mechanism that keeps the slow-learning weights anchored to past experience. We will examine these methods in detail in a later chapter.

Contrast this with a modular architecture where each task uses a dedicated subset of parameters. Task 1's parameters are never touched when learning task 2, so forgetting is impossible by construction. But pure modularity trades away parameter efficiency and generalization: you cannot share representations across tasks, which wastes capacity and prevents the transfer of general knowledge. A model with separate parameter blocks for every task it has ever seen would scale linearly in size with the number of tasks, which is not sustainable.

The ideal architecture lies somewhere between these extremes: enough sharing to enable transfer and compression, enough isolation to prevent destructive interference. Architectures that achieve this include progressive neural networks, which add new columns for new tasks while keeping old columns frozen; and dynamic sparse networks, which grow new pathways while preserving old ones. The tension between sharing and isolation at the architectural level mirrors the plasticity-stability dilemma at the optimization level; they are the same problem viewed from different vantage points.

Transformer architectures, which underpin modern language models, are particularly susceptible to catastrophic forgetting in certain parts of the network. The attention layers encode context-dependent patterns, but the feed-forward layers encode task-specific knowledge. Research on mechanistic interpretability has found that feed-forward layers in transformers act as key-value stores where factual associations are encoded. When you fine-tune a language model on new factual content, the gradient updates to these feed-forward layers overwrite some of the previously stored associations.

The attention layers present a different problem. Multi-head attention learns query and key projections that determine which tokens attend to which other tokens. Fine-tuning on a new task with a different attention structure can shift these projections in ways that change how the model processes all inputs, not just those from the new task. The shared architecture means that a single gradient step on a new task touches the same matrices that support every previously learned capability.

This observation has led to parameter-efficient fine-tuning methods (which you studied in the PEFT chapters) being adopted for efficiency and as an implicit catastrophic forgetting mitigation strategy. Methods like LoRA, which learn small low-rank updates rather than modifying all weights, naturally limit the extent of interference because the core pre-trained weights remain frozen. The new task is encoded in a small, isolated set of parameters, while the original model's capabilities are preserved in the frozen backbone.

Forgetting is measured through several evaluation metrics. Let $a_{k,j}$ denote the accuracy of the model on task $j$ after training on task $k$ (where $k \geq j$). The key metrics are:

Backward Transfer (BWT) measures how much learning on later tasks hurts earlier tasks:

$$\text{BWT} = \frac{1}{N-1} \sum_{k=2}^{N} (a_{N,k-1} - a_{k-1,k-1})$$

where:

• $a_{N,k-1}$: accuracy on task $k-1$ after training through all $N$ tasks (final model)
• $a_{k-1,k-1}$: accuracy on task $k-1$ immediately after training on it (before any subsequent tasks)
• A negative BWT value indicates catastrophic forgetting; more negative means more forgetting
• A BWT of zero means later tasks cause no degradation on earlier tasks

Forward Transfer (FWT) measures how learning earlier tasks helps on new tasks:

$$\text{FWT} = \frac{1}{N-1} \sum_{k=2}^{N} (a_{k-1,k} - b_k)$$

where:

• $a_{k-1,k}$: accuracy on task $k$ when evaluated before training on it (using the model from the previous task)
• $b_k$: accuracy on task $k$ with a randomly initialized model (baseline)
• Positive FWT means prior tasks helped generalize to new tasks; negative FWT means negative transfer

Average Accuracy across all tasks after completing all training:

$$\text{Avg} = \frac{1}{N} \sum_{k=1}^{N} a_{N,k}$$

Together, these metrics paint a complete picture of a continual learner: does it retain what it learned (BWT near zero), does it benefit from accumulated knowledge when encountering new tasks (positive FWT), and what is its overall competence across all tasks (Avg)?

Understanding all three metrics together matters because they can tell conflicting stories. A method with good average accuracy might achieve it by learning recent tasks very well at the cost of severe forgetting of early tasks, hiding the forgetting behind the recency of recent tasks in the average. Inspecting BWT separately exposes this. Conversely, a method with a BWT near zero might achieve stability by learning very slowly (low plasticity), resulting in poor per-task performance even immediately after training, which would show up as low average accuracy. A complete evaluation reports all three.