Continual Learning: Catastrophic Forgetting and Scenarios

Every model you have studied so far in this book learned from a fixed dataset. You gathered data, split it into training and validation sets, trained until the model converged, and deployed. When the world changed or new data arrived, the standard response was to retrain from scratch. This works reasonably well when data fits comfortably in memory and training is cheap. But the real world rarely cooperates on either count.

Imagine a language model deployed to assist customer support agents. In January, users ask about product features. By March, a new product launches and brings new terminology. By June, a competitor event shifts how customers describe their problems. If you retrain the model from scratch each time, you spend enormous compute, require large amounts of old data to remain on hand, and introduce deployment lag. If you continue training on new data alone, something alarming happens: the model forgets what it learned before.

This forgetting is not a minor nuisance. It is a fundamental and well-characterized failure mode called catastrophic forgetting. Understanding it, formalizing it, and eventually mitigating it is the subject of the continual learning research area. This chapter establishes the problem: what continual learning means, why catastrophic forgetting occurs at the level of gradient descent and parameter space, how different learning scenarios shape the difficulty of the problem, and what the specific failure modes look like in practice. Subsequent chapters in this part of the book will develop the tools to address these problems.

Continual learning, also called lifelong learning or sequential learning, refers to the setting where a model must learn from a non-stationary stream of tasks or data distributions over time. The model encounters each task or data batch sequentially, and it typically cannot revisit earlier data once it has moved on to the next task.

The phrase "without access to all historical data simultaneously" is the key constraint. If you could always train on all data seen so far, you would simply retrain the model on the growing combined dataset. Continual learning becomes a problem precisely because this option is unavailable, whether due to storage limits, privacy constraints, regulatory requirements, or simply the volume of data that accumulates over a long deployment lifetime.

The key tension is between two competing objectives:

• Plasticity: the ability to learn new information and adapt to new tasks
• Stability: the ability to retain previously learned knowledge

An ideal continual learner would be fully plastic (rapidly absorbing new information) and fully stable (never degrading on past tasks). In practice, these objectives conflict directly. Increasing plasticity means the model's parameters shift more aggressively toward new data, which naturally overwrites gradients from old data. Increasing stability means the model's parameters resist change, which slows learning on new tasks. This tradeoff is called the plasticity-stability dilemma, and it is the central challenge in continual learning.

The dilemma is not unique to machine learning. Cognitive science has grappled with the same tension for decades, trying to understand how the human brain acquires new information across a lifetime without erasing old memories. The neural network model of the brain was initially expected to suffer catastrophically from sequential learning, just as artificial neural networks do. The brain's solution involves a distributed memory architecture with distinct subsystems that operate at different learning rates, a design that has directly inspired some of the most effective continual learning algorithms.

Retraining from scratch is conceptually simple but practically expensive for several reasons:

• Compute cost: Training large models takes days or weeks on expensive hardware. Retraining for every incremental update is prohibitive at scale. A model with hundreds of billions of parameters trained on trillions of tokens costs millions of dollars to train once. Doing this every time new data arrives is economically untenable.
• Data retention: Retraining requires keeping all historical data. For large-scale language models, this means storing terabytes or petabytes of text indefinitely, which raises substantial storage, privacy, and regulatory costs. GDPR and similar regulations may require organizations to delete user data upon request, making indefinite retention legally problematic.
• Deployment latency: The gap between new data arriving and an updated model being deployed can stretch from hours to weeks when full retraining is required. For time-sensitive applications, such as financial news analysis or crisis response systems, this latency is operationally unacceptable.
• Catastrophic compute waste: Full retraining throws away all the gradient work done in previous training runs. Every iteration of a new training run spends compute rediscovering representations that the previous model already encoded. Incremental updates could in principle converge much faster if they could build on existing representations rather than relearning them from scratch.
• Biological implausibility: Humans do not forget how to ride a bicycle when learning to drive a car. They do not forget their native language when learning a second one. If the goal is human-like general intelligence, models should similarly accumulate knowledge across tasks rather than resetting whenever something new is learned.

These pressures motivate a different approach: learning from new data incrementally, updating only what needs to change, while preserving what was already learned. This is the continual learning program.

Let us formalize the continual learning problem. Suppose we have a sequence of tasks $\mathcal{T} = \{T_1, T_2, \ldots, T_N\}$, where each task $T_k$ is associated with a dataset $\mathcal{D}_k = \{(x_i^{(k)}, y_i^{(k)})\}_{i=1}^{n_k}$ drawn from distribution $p_k(x, y)$.

A model with parameters $\theta$ is trained sequentially. After training on task $T_k$, the model has parameters $\theta_k$. The continual learning objective is:

where:

• $\theta_N$: the model parameters after training on all $N$ tasks sequentially
• $\mathcal{L}_k(\theta_N)$: the loss on task $k$ evaluated with the final parameters, measuring how well the model has retained task $k$'s knowledge
• $N$: the total number of tasks encountered over the model's lifetime

The constraint is that when training on task $T_k$, the model typically has access only to $\mathcal{D}_k$, not to any $\mathcal{D}_j$ for $j < k$. This data constraint is what makes the problem hard. Without it, you could simply minimize the sum of losses jointly by training on all data simultaneously.

Notice what the objective asks for: it wants the final parameters $\theta_N$ to perform well on every task rather than only the last one. This is categorically different from the standard supervised learning objective, which only cares about performance on the current training distribution. The continual learning objective asks the model to remain competent across its entire history of tasks, even though the gradient updates during training on task $k$ will only ever see $\mathcal{D}_k$.

The gap between this objective and the data constraint is where all the difficulty lives. The optimizer has no signal from $\mathcal{D}_1, \ldots, \mathcal{D}_{k-1}$ when it updates $\theta$ during training on task $k$, so it has no direct reason to preserve performance on those earlier tasks. Any preservation must come from architectural constraints, memory mechanisms, or regularization that encode information about the past into the update rule itself.

The spectrum in @fig-plasticity-stability maps directly onto design choices you make when implementing continual learning. A high regularization weight on parameter changes produces a model near the stability end. A small replay buffer with infrequent past-data rehearsal produces a model near the plasticity end. Most practical methods live somewhere in the middle, and the right position depends on how frequently tasks change and how important it is to retain older versus newer knowledge.

The phenomenon of catastrophic forgetting (also called catastrophic interference) was first described by McCloskey and Cohen in 1989, and later studied extensively by Ratcliff (1990) and French (1999). When a neural network learns new information, the gradient updates that encode that new information overwrite the weights that encoded previous information. The overwriting is not gradual or selective; it is typically rapid and near-total. This is the "catastrophic" part of the name.

What makes this so striking is the contrast with biological memory. When you learn a new recipe, you do not forget all previous recipes. When you learn a new song, you do not lose the ability to play previously learned songs. Human memory is susceptible to interference, but the degree of interference is nowhere near what artificial neural networks experience. A neural network trained sequentially on two tasks can lose essentially all performance on the first task within dozens of gradient steps on the second, even when the two tasks appear superficially similar to a human observer.

To understand why forgetting happens, consider what gradient descent does. When training on task $T_1$, the optimizer moves the parameters $\theta$ to minimize $\mathcal{L}_1(\theta)$. The loss surface for task 1 has a basin (a region of low loss) that the optimizer settles into. The parameters at this optimum are tuned precisely to the features of task 1's data distribution: the weight magnitudes and directions encode the statistical regularities of that particular task.

When training then shifts to task $T_2$, the optimizer now minimizes $\mathcal{L}_2(\theta)$, which has a completely different loss surface. Gradient descent moves parameters toward the minimum of $\mathcal{L}_2$, regardless of where the minimum of $\mathcal{L}_1$ was. Unless the two loss surfaces happen to share a low-loss region, moving toward the task 2 minimum means moving away from the task 1 minimum. The model is not "deciding" to forget task 1; it is simply following gradients that have no knowledge of or concern for task 1 performance.

This can be visualized in parameter space. Imagine two elliptical basins in a 2D parameter landscape. The optimizer first descends into basin 1. Then it starts descending toward basin 2. If these basins are separated, the model's parameters end up in basin 2, far from where they performed well on task 1. The model has forgotten task 1, not because of any design failure, but because that is exactly what unconstrained gradient descent on the task 2 objective will do.

In high-dimensional parameter spaces (billions of parameters in large language models), the situation is more complex but the same principle applies. Each task carves out a region of parameter space where its performance is high. Sequential gradient updates do not respect the boundaries of previous tasks' performance regions. The optimizer is blind to them, because the loss function it is minimizing does not include any term that penalizes moving outside those regions.

The two basins in @fig-loss-landscape-forgetting do not overlap. That is the crux of the problem. For a simple two-task, two-parameter toy example, you can imagine a joint basin that satisfies both tasks, and indeed in very high-dimensional spaces such overlap does exist. But finding that overlap through gradient descent on only one task's loss is not guaranteed, and in practice the optimizer takes the path of least resistance along the gradient, which is the path toward the current task's minimum and away from everything else.

The forgetting problem is exacerbated by how neural networks distribute their representations. In a fully connected network, every weight participates in encoding information about every input. When you learn task 2, updates to shared weights inevitably disrupt the encoding of task 1's features. There is no structural boundary that separates "task 1 weights" from "task 2 weights"; the same matrix entries that encode syntax for one task also encode vocabulary statistics for another. Gradient descent on task 2 updates all of them simultaneously.

This is fundamentally different from how biological brains handle sequential learning. Neuroscience research suggests that the hippocampus acts as a fast-learning memory buffer that consolidates new experiences rapidly, while the neocortex learns slowly and represents general patterns extracted over many episodes. This complementary learning systems theory, proposed by McClelland, McNaughton, and O'Reilly in 1995, posits that the brain's resistance to catastrophic forgetting comes precisely from this two-speed architecture. New memories are held in a volatile fast store while gradually transferred to a stable slow store, preventing the sudden interference that plagues single-speed learners.

This biological insight directly inspired experience replay methods in continual learning, which maintain a small buffer of past examples and interleave them with new data during training. The replay buffer functions like a simplified hippocampus, giving a fast retrieval mechanism that keeps the slow-learning weights anchored to past experience. We will examine these methods in detail in a later chapter.

Contrast this with a modular architecture where each task uses a dedicated subset of parameters. Task 1's parameters are never touched when learning task 2, so forgetting is impossible by construction. But pure modularity trades away parameter efficiency and generalization: you cannot share representations across tasks, which wastes capacity and prevents the transfer of general knowledge. A model with separate parameter blocks for every task it has ever seen would scale linearly in size with the number of tasks, which is not sustainable.

The ideal architecture lies somewhere between these extremes: enough sharing to enable transfer and compression, enough isolation to prevent destructive interference. Architectures that achieve this include progressive neural networks, which add new columns for new tasks while keeping old columns frozen; and dynamic sparse networks, which grow new pathways while preserving old ones. The tension between sharing and isolation at the architectural level mirrors the plasticity-stability dilemma at the optimization level; they are the same problem viewed from different vantage points.

Transformer architectures, which underpin modern language models, are particularly susceptible to catastrophic forgetting in certain parts of the network. The attention layers encode context-dependent patterns, but the feed-forward layers encode task-specific knowledge. Research on mechanistic interpretability has found that feed-forward layers in transformers act as key-value stores where factual associations are encoded. When you fine-tune a language model on new factual content, the gradient updates to these feed-forward layers overwrite some of the previously stored associations.

The attention layers present a different problem. Multi-head attention learns query and key projections that determine which tokens attend to which other tokens. Fine-tuning on a new task with a different attention structure can shift these projections in ways that change how the model processes all inputs, not just those from the new task. The shared architecture means that a single gradient step on a new task touches the same matrices that support every previously learned capability.

This observation has led to parameter-efficient fine-tuning methods (which you studied in the PEFT chapters) being adopted for efficiency and as an implicit catastrophic forgetting mitigation strategy. Methods like LoRA, which learn small low-rank updates rather than modifying all weights, naturally limit the extent of interference because the core pre-trained weights remain frozen. The new task is encoded in a small, isolated set of parameters, while the original model's capabilities are preserved in the frozen backbone.

Forgetting is measured through several evaluation metrics. Let $a_{k,j}$ denote the accuracy of the model on task $j$ after training on task $k$ (where $k \geq j$). The key metrics are:

Backward Transfer (BWT) measures how much learning on later tasks hurts earlier tasks:

where:

• $a_{N,k-1}$: accuracy on task $k-1$ after training through all $N$ tasks (final model)
• $a_{k-1,k-1}$: accuracy on task $k-1$ immediately after training on it (before any subsequent tasks)
• A negative BWT value indicates catastrophic forgetting; more negative means more forgetting
• A BWT of zero means later tasks cause no degradation on earlier tasks

Forward Transfer (FWT) measures how learning earlier tasks helps on new tasks:

where:

• $a_{k-1,k}$: accuracy on task $k$ when evaluated before training on it (using the model from the previous task)
• $b_k$: accuracy on task $k$ with a randomly initialized model (baseline)
• Positive FWT means prior tasks helped generalize to new tasks; negative FWT means negative transfer

Average Accuracy across all tasks after completing all training:

Together, these metrics paint a complete picture of a continual learner: does it retain what it learned (BWT near zero), does it benefit from accumulated knowledge when encountering new tasks (positive FWT), and what is its overall competence across all tasks (Avg)?

Understanding all three metrics together matters because they can tell conflicting stories. A method with good average accuracy might achieve it by learning recent tasks very well at the cost of severe forgetting of early tasks, hiding the forgetting behind the recency of recent tasks in the average. Inspecting BWT separately exposes this. Conversely, a method with a BWT near zero might achieve stability by learning very slowly (low plasticity), resulting in poor per-task performance even immediately after training, which would show up as low average accuracy. A complete evaluation reports all three.

The numbers tell the story clearly. After training on task 1, the model achieves strong accuracy on that task. Then, when we train the same model on task 2 without any mechanism to protect task 1's parameters, accuracy on task 1 collapses. The backward transfer metric is significantly negative, confirming catastrophic forgetting. Meanwhile, the model learns task 2 well. This is the classic signature of sequential training without any continual learning technique applied.

Here, "learning task 2 well" means redirecting capacity rather than solving a harder problem. It has redirected all its representational capacity toward the new task by forgetting task 1. The total amount of useful work the model can do has not increased; it has shifted. The goal of continual learning is to allow that total useful work to accumulate rather than shift.

Not all continual learning problems are alike. The research community has settled on three canonical scenarios that differ in what information is available during training and inference. Understanding these scenarios is important because different methods perform differently under each, and real-world applications map onto different scenarios depending on context.

The scenarios were systematically analyzed by van de Ven and Tolias (2019), who formalized the distinctions and highlighted that many papers claiming to solve continual learning were only solving the easiest scenario while presenting results in ways that obscured this. Knowing which scenario you are in determines which methods are applicable and what performance you should reasonably expect.

The three scenarios all share the same sequential training setup: the model sees tasks one at a time and cannot revisit previous data. They differ in how much task identity information is available and whether the output space changes between tasks.

In task-incremental learning, the model is given an explicit task identity at both training and test time. When you present an input for inference, you also tell the model which task it came from. This is the simplest and most permissive scenario.

Because the model always knows the current task, it can use separate output heads, one per task, and route inputs to the correct head based on the task identity. The learning challenge is therefore confined to the shared feature representation in the backbone network: can the network learn to extract useful features for all tasks without the early layers' representations being disrupted by each new task?

In natural language processing, task-incremental learning corresponds to settings like training a model sequentially on sentiment classification, then named entity recognition, then question answering, where at inference time the system always knows which of these tasks it is performing. An enterprise NLP pipeline that applies different models to different document types and knows the document type in advance is effectively task-incremental. The system selects the right task head based on the document category signal, so the only challenge is maintaining the shared encoder's quality across sequential training.

Task-IL is the least challenging scenario because the task identity signal eliminates a large source of ambiguity. The model never needs to figure out which task an input belongs to; it is told. This allows the model to maintain completely separate parameter subsets for each task's output, sidestepping the class confusion problem entirely. However, task-IL is also the least realistic in fully general settings, because real-world deployments often cannot guarantee that task identity will always be available at inference time.

The practical relevance of task-IL is highest in systems where the task is explicitly signaled by the application context. A voice assistant that knows whether it is in "navigation mode," "music mode," or "calendar mode" effectively operates in a task-IL setting. A document processing pipeline that knows whether it is handling a contract, an invoice, or a report is similar. For these cases, task-IL methods are appropriate and can achieve near-oracle performance with relatively simple techniques.

In domain-incremental learning, the task structure (the type of problem and the output space) remains the same across all tasks, but the input distribution shifts. The model must solve the same kind of problem on inputs from different distributions. The model does not receive task identity at test time.

Think of training a language model on news articles from 2020, then fine-tuning on news from 2021, then 2022. The task (next-token prediction or text classification) does not change, but the vocabulary, topics, and stylistic patterns shift as language evolves and new events occur. At inference time, you cannot tell the model which year the test document came from; it must handle all of them with a single set of parameters and a single output head.

Domain-IL is harder than task-IL because without knowing the domain, the model must develop representations that generalize across domains rather than routing to task-specific heads. There is only one output structure, and gradient updates for the new domain directly affect the weights used for the old domain. The model cannot fall back to a separate head for old inputs.

The domain shift in Domain-IL can take many forms. It might be temporal, as in the news example above, where language patterns evolve over time. It might be stylistic, such as training on academic text and then fine-tuning on social media. It might be topical, such as a medical question-answering system that must handle cardiology questions after being trained primarily on oncology. In all these cases, the input distribution changes but the output structure does not. The model must generalize across distributions that it has not seen jointly.

In the medical imaging literature, domain-IL corresponds to training a diagnostic model on images from one hospital scanner, then adapting it to a different scanner with different calibration and imaging characteristics, while maintaining accuracy on both scanner types without being told which scanner produced a given test image. This is a practically important scenario for deploying medical AI systems across healthcare institutions with heterogeneous equipment.

Class-incremental learning is the most challenging scenario. Here, the set of output classes grows over time, and the model must perform classification across all classes seen so far, without knowing which subset of classes the current input belongs to.

Consider training an image classifier first on cats and dogs (task 1), then on birds and fish (task 2), then on horses and rabbits (task 3). At test time, you present an arbitrary image and ask the model to classify it into one of all six classes. The model must remember the features of cats and dogs and correctly distinguish them from birds, fish, horses, and rabbits, even though all six classes were never seen together during training.

For language models, class-IL corresponds to training a text classifier on an expanding set of categories. You might start with spam/not-spam, then add categories for phishing, promotional, and transactional content, and the model must correctly classify any email into the right category using all labels seen so far. Each new batch of training data contains only examples from the new categories, so the model must somehow preserve its ability to recognize the old categories as well.

Class-IL is hardest for two interconnected reasons. First, the model must remember old classes without access to old data. If the only examples of cats and dogs the model ever sees come from task 1, but task 1's data is gone by the time the model encounters task 3, then the task-3 gradient updates will push the classifier weights in directions that improve fish versus birds but may simultaneously degrade cats versus dogs. Second, the model must calibrate its confidence appropriately across an expanding output space. A model trained only on the most recent classes will exhibit extreme recency bias, classifying almost everything as belonging to the most recently learned classes, because its decision boundaries have been tuned to those classes without any pressure to maintain separation from earlier classes.

The recency bias in class-IL is particularly pernicious because it is invisible within the current task's training data. If you evaluate the model only on the classes it just learned, it will look excellent. The forgetting only becomes apparent when you evaluate it on the full set of classes it is supposed to handle, including the ones from earlier tasks. This is why a proper class-IL evaluation must always test on all classes seen so far rather than only the most recent ones.

The importance of distinguishing these scenarios cannot be overstated. A method that achieves 95% accuracy in a task-IL benchmark and 45% in a class-IL benchmark on the same dataset is not a general continual learner; it works in a specific and favorable condition. Results that do not specify which scenario was used are difficult to interpret or compare, and comparisons between papers using different scenarios without acknowledging this are misleading.

When reading the continual learning literature, always check which scenario is assumed. A surprisingly large number of influential papers evaluated their methods in the task-IL setting while presenting the results as solutions to continual learning generally. Van de Ven and Tolias' 2019 survey brought this issue to wider attention and substantially improved the field's evaluation standards. Even so, you should check carefully before drawing conclusions from any continual learning paper.

Beyond the high-level forgetting phenomenon, continual learning research has identified several specific failure modes that arise under different conditions. Understanding these helps you diagnose what is going wrong in a deployment and points toward the appropriate solution family.

When a model is trained sequentially without any memory mechanism, its outputs become strongly biased toward the distribution of the most recent task. If the final training batch contained mostly examples from class A, the model's parameters will reflect that distribution, and it will classify a disproportionate fraction of test inputs as class A, even when those inputs should be class B or C.

Recency bias is especially severe in class-incremental learning. The model's classifier output layer learns weights that maximize accuracy on the current task, ignoring calibration with respect to previously learned classes. The softmax logits for old classes will be lower on average than for new classes, not because the model has "decided" old classes are less common, but because the output weights for old classes have drifted under gradient pressure from new class training and are no longer calibrated to produce competitive logit values.

The practical consequence of recency bias is that a model appears to perform well on recent data but fails systematically on older categories. In a production system, this can manifest as a classifier that worked well on a pilot dataset (representing all classes) but degrades on an expanding production dataset (where the most recently added categories dominate). Monitoring class-specific precision and recall, rather than just aggregate accuracy, is essential for detecting recency bias in deployment.

Even when explicit classification performance is maintained at the top-level output, the internal representations learned by the model can drift significantly during sequential training. If a downstream component or task was trained on the model's intermediate representations from an earlier stage, that downstream component may fail even if the model's top-level task performance has been preserved.

This matters particularly in multi-task NLP systems where a shared encoder feeds multiple task-specific decoders. If the encoder drifts during fine-tuning on new tasks, all previously fine-tuned decoders may degrade simultaneously. You might observe that the model's accuracy on task A has not changed (because you are using the correct output head for task A), but a semantic similarity model that was calibrated against the encoder's earlier representations now produces meaningless similarity scores.

Representation drift is especially relevant for embedding-based retrieval systems. If you fine-tune a sentence encoder on a new dataset of documents, the embedding space shifts. Old documents indexed against the previous embedding space may no longer retrieve correctly when the query encoder has been updated. The index needs to be rebuilt from scratch, which may be expensive. This problem, sometimes called the index staleness problem, is a practical manifestation of representation drift in production.

Monitoring representation drift requires measuring the similarity between old and new representations of the same inputs, not just task accuracy. Techniques like centered kernel alignment (CKA) can measure how much two neural network representations agree with each other across inputs, and tracking CKA between pre-fine-tuning and post-fine-tuning representations reveals how much the embedding space has shifted, independently of task performance.

In domain-incremental and class-incremental learning, the model may learn to perform well on individual tasks in isolation but fail when inputs from different tasks or domains are presented together. The model has not learned to distinguish which domain an input came from, so it applies the wrong set of learned patterns.

Task confusion is more than a label assignment problem. It reveals that the model has failed to build a unified representation space that can disambiguate across all domains or classes. In a continual learning setting where each task arrives with its own distribution, the model may develop domain-specific shortcuts that work well within a task but break down when exposed to the full mixture. A language model trained sequentially on formal legal text and then casual social media posts, for example, might apply legal-style disambiguation heuristics to casual language examples at inference time, generating stilted or incorrect interpretations.

This is distinct from recency bias, though the two can co-occur. Recency bias means the model overestimates the probability of recent classes regardless of input features. Task confusion means the model applies the wrong feature-to-output mapping based on misidentifying the distribution of the input. Both lead to errors, but the underlying causes and solutions differ. Recency bias is a calibration problem that can sometimes be addressed by output weight rescaling. Task confusion is a representation problem that requires learning features that generalize across distributions.

The solution to task confusion is not simply to train longer or on more data within each task. It requires building representations that remain stable under distributional variation, which is precisely what methods like domain-adversarial training and invariant risk minimization aim to provide.

At the optimization level, catastrophic forgetting manifests as gradient interference: the gradients computed on new task data point in directions that conflict with the directions needed to maintain performance on old tasks. In parameter space, you cannot simultaneously satisfy the gradient constraints of all past tasks, and without explicit mechanisms to reconcile these conflicts, the most recent gradients dominate.

To see this concretely, suppose the gradient of the loss for task 1, evaluated at the current parameters, is the vector $g_1$, and the gradient for task 2 is $g_2$. Standard stochastic gradient descent updates parameters by moving in the direction of $-g_2$ (minimizing task 2 loss). If $g_1 \cdot g_2 < 0$ (the two gradients are in conflicting directions), then the step that reduces task 2 loss will simultaneously increase task 1 loss. This is gradient interference in its most direct form.

In high-dimensional spaces, full interference is unlikely: with millions of parameters, there are many directions that improve task 2 performance while not harming task 1. But gradient descent does not search for such directions; it blindly follows the task 2 gradient. The task 1 gradient information is simply discarded once task 1's data is no longer available.

The inner product $g_1 \cdot g_2$ tells you the cosine alignment between the gradients for the two tasks. When this quantity is positive, the two tasks are aligned: a step toward task 2's minimum simultaneously helps task 1 as well. When it is zero, the tasks are orthogonal in gradient space and the step is neutral. When it is negative, the tasks actively conflict, and any progress on task 2 comes at the expense of task 1. The degree of gradient interference depends on how similar the two tasks are at the representation level; tasks that share useful features tend to have more positively aligned gradients.

The landmark work by Lopez-Paz and Ranzato (2017) formalized this gradient perspective with the Gradient Episodic Memory (GEM) method, which stores a small buffer of past task examples and constrains new task gradient updates to not increase the loss on stored examples. Specifically, GEM requires that the inner product $g_{new} \cdot g_{past} \geq 0$ for all past tasks, projecting the gradient onto a feasible cone if this condition is violated. This ensures that the new update is at least neutral with respect to past tasks, preventing gradient interference while still allowing learning.

Later work by Saha et al. (2021) extended this idea with subspace projections, finding directions in parameter space that simultaneously improve the new task without degrading stored representations of old tasks. These gradient-based perspectives have been highly influential because they tie the forgetting problem directly to the mechanics of optimization rather than treating it as a black-box empirical phenomenon. If you understand gradient interference, you can design update rules that explicitly avoid it.

Let us build a clearer picture of how forgetting unfolds by simulating the full sequential training process with accuracy tracking at each step. This walkthrough demonstrates BWT, FWT, and the progression of performance across tasks in a controlled, observable setting.

The plot in @fig-catastrophic-forgetting-curves makes forgetting viscerally clear. At each dashed task boundary, at least one earlier task loses accuracy abruptly. Task 1 retains some of what it learned after Task 2 but degrades further after Task 3, while Task 2 falls from near-perfect accuracy to almost zero as soon as Task 3 training begins. The damage occurs at the distribution shift rather than as a slow decay, which is the "catastrophic" aspect of catastrophic forgetting.

Notice also what happens to the future tasks before training on them begins. Their accuracy hovers around chance level during earlier task phases, because the model has not been exposed to their data at all. Once training on each task begins, accuracy rises rapidly, which represents normal learning rather than anything unusual. The problem is entirely in what happens to earlier tasks when this learning takes place.

The catastrophic forgetting problem is especially acute for large language models. Consider a model trained on a massive corpus that captures language patterns, factual knowledge, and reasoning abilities across countless domains. Performing any subsequent fine-tuning on a narrow task, such as instruction following or domain adaptation, risks catastrophic forgetting of the broad capabilities that make the model valuable in the first place.

This is not a hypothetical concern. Early empirical studies of LLM fine-tuning found that extensive fine-tuning on task-specific data could substantially degrade performance on general benchmarks. A model fine-tuned aggressively for code generation might lose some of its conversational ability. A model fine-tuned on customer support conversations might lose factual recall on topics that appear rarely in support data. The community initially addressed this through limited fine-tuning (few epochs, lower learning rates) and later through methods like RLHF with careful KL-divergence penalties to prevent the fine-tuned model from drifting too far from the reference model.

The KL-divergence penalty in RLHF deserves special note because it is essentially a continual learning technique, even if it is rarely described that way. By penalizing the updated policy for deviating too far from the reference policy, RLHF prevents catastrophic forgetting of the base model's language competence during reward optimization. The reference model is an implicit memory of the pre-fine-tuning distribution. This is analogous to EWC (Elastic Weight Consolidation), a regularization-based continual learning method that penalizes changes to parameters that were important for previous tasks.

The problem compounds when LLMs need to be updated continuously. New events happen, new facts emerge, and the model's knowledge becomes stale. Retraining from scratch to incorporate new knowledge is prohibitively expensive. Continual fine-tuning without forgetting mechanisms means the model gradually loses its general capabilities. Researchers are actively developing techniques to enable LLMs to incorporate new knowledge while preserving existing capabilities, a problem sometimes called knowledge editing or model editing in the LLM literature.

Building on the sequence-to-sequence architectures and attention mechanisms studied in earlier parts of this book, large transformer-based language models carry the same fundamental vulnerability: all knowledge is stored distributedly across the weight matrices, and gradient updates for new tasks do not discriminate between weights that encode new information and weights that encode old knowledge. The distributed, entangled nature of knowledge in neural networks is both their greatest strength (it enables generalization and abstraction) and their greatest weakness for continual learning (it makes knowledge hard to update selectively).

One active research direction treats the continual learning problem in language models as a knowledge editing problem: rather than fine-tuning the entire model on new data, you want to surgically update specific facts or capabilities while leaving the rest of the model unchanged. Methods like ROME (Rank-One Model Editing) and MEMIT identify the specific feed-forward layers and neurons that store particular factual associations, then perform targeted updates that change only those associations without affecting other knowledge.

This approach has important advantages from a continual learning perspective. Targeted edits cause minimal representation drift compared to global fine-tuning. They do not require large datasets of new examples; a single fact update might require only a handful of training examples. And they are reversible: if an edit turns out to be wrong, you can undo it without retraining the whole model.

However, knowledge editing also has limitations. The assumption that facts are localized in specific network components is a simplification. Some knowledge is distributed across many components and cannot be cleanly edited. Editing one fact can have unintended side effects on related knowledge. And the approach does not scale easily to large-scale continual updates where hundreds or thousands of facts change simultaneously.

Instruction-tuned models face a particularly interesting form of the catastrophic forgetting problem. These models have been fine-tuned to follow instructions, have conversations, and refuse harmful requests. Any subsequent fine-tuning, whether for domain adaptation or task specialization, risks degrading the instruction-following capability itself.

This creates a tiered forgetting problem. The base pre-training capabilities (language modeling, factual recall, reasoning) can be degraded by fine-tuning. The instruction-tuning capabilities (following instructions, formatting responses, maintaining safety behaviors) can be degraded by subsequent fine-tuning on top of the instruction-tuned model. Each fine-tuning stage is a new "task" in the continual learning sense, and each stage risks overwriting the achievements of earlier stages.

A practical illustration: if you take an instruction-tuned language model and fine-tune it on a dataset of technical documents, you might find that the model's instruction-following ability degrades, it starts producing responses in the style of technical documentation rather than conversational text, or it loses some of its safety behaviors. These are all forms of catastrophic forgetting, operating at different levels of the model's learned behavior hierarchy.

Let us work through a concrete numerical example to solidify the metrics and their interpretation. Suppose we have two tasks and a model whose accuracy evolves as follows:

After training on task 1: accuracy on task 1 is 0.88. After training on task 2: accuracy on task 1 drops to 0.52, accuracy on task 2 is 0.84.

We can compute:

where:

• $a_{2,1}$: accuracy on task 1 after training on task 2 (final evaluation)
• $a_{1,1}$: accuracy on task 1 immediately after training on it

A BWT of $-0.36$ means that completing training on task 2 caused a 36-percentage-point drop in task 1 performance. This is severe forgetting. Any BWT worse than $-0.10$ in a real deployment would be a serious quality problem for most applications.

For forward transfer, suppose before training on task 2 at all, the model (from its task 1 training) already achieves 0.62 on task 2, and a random baseline achieves 0.50:

where:

• $a_{1,2}$: accuracy on task 2 evaluated with the task-1-trained model, before any task 2 training
• $b_2$: random baseline accuracy on task 2

A positive FWT of 0.12 means that learning task 1 provided a 12-percentage-point head start on task 2, suggesting positive transfer of shared representations. This is an encouraging sign: the two tasks share enough structure that learning one helps with the other. In language model settings, positive forward transfer is common because many NLP tasks share underlying linguistic structure. A model that learned to recognize sentiment may generalize somewhat to detecting subjectivity even before seeing subjectivity labels.

The average final accuracy across both tasks is:

This average is pulled down substantially by the forgetting on task 1. A good continual learning method would aim to keep average accuracy closer to the individual task peaks (around 0.86 in this example) by preventing BWT from becoming so negative. The ideal outcome would be something like: task 1 accuracy stays at 0.87 after task 2 training, and task 2 accuracy reaches 0.84 as before. Average accuracy would then be 0.86, and BWT would be approximately $-0.01$: essentially zero forgetting with full learning.

The output confirms our manual calculation. The combination of negative BWT and positive FWT tells a coherent story: the model is learning tasks in a way that enables some positive transfer (the tasks share useful features), but sequential gradient updates are still destructive enough to cause severe forgetting. A continual learning method that could preserve the positive FWT while bringing BWT close to zero would be a substantial improvement.

This pattern, strong forward transfer combined with severe backward interference, is common in practice and suggests that the solution should focus on protecting previously learned parameters rather than on how new tasks are initially learned. The new task learning itself is working well (FWT is positive); what needs improvement is the mechanism that prevents this learning from overwriting old knowledge.

Understanding the continual learning problem is only the beginning. Several practical realities make it harder than the formal problem statement suggests.

The three-scenario taxonomy (Task-IL, Domain-IL, Class-IL) provides a clean theoretical framework, but real deployments rarely map cleanly onto one scenario. In production, you might know the task for some inputs but not others, or the task distribution might shift gradually rather than in discrete steps. Evaluating continual learning methods against benchmark scenarios can give misleadingly optimistic results when the actual deployment scenario is harder.

Practitioners should be skeptical of any continual learning result that does not clearly specify the scenario and evaluation protocol. The gap between task-IL and class-IL performance for the same method can be 30 to 50 percentage points on standard benchmarks. Comparing a task-IL result to a class-IL baseline is not meaningful, yet this comparison is easy to make accidentally if the scenario is not clearly specified.

Beyond the scenario ambiguity, benchmark datasets used for continual learning evaluation often do not reflect the complexity of real sequential data distributions. Datasets like Split-MNIST (which divides MNIST classes into five pairs of classes presented sequentially) are highly controlled and may not reveal failures that appear in messier real-world distributions. Results from clean benchmark settings should be taken as optimistic estimates of real-world performance.

Most continual learning methods require storing some form of memory: a replay buffer of past examples, a set of compressed past task representations, or parameter snapshots that can be used to constrain updates. The size of this memory and the compute cost of maintaining it must be weighed against the forgetting it prevents. For very large models, even a small percentage of parameters dedicated to memory is expensive in absolute terms.

There is no free lunch in continual learning: every method that effectively prevents forgetting does so by dedicating some resource (memory, compute, capacity, or constrained parameter space) to preserving past knowledge. The design question is always which resource to spend, and how much of that resource purchases how much forgetting prevention. The relationship is not linear; a very small replay buffer may prevent most forgetting for most tasks, while a larger buffer provides diminishing returns.

For language models specifically, the memory costs of replay-based methods are substantial. Storing representative examples from previous fine-tuning datasets requires either keeping the original data (which may not be possible for privacy reasons) or generating synthetic examples that capture the distribution (which adds generation cost and introduces approximation error). Parameter-efficient methods like LoRA that avoid touching the base model weights sidestep some of these issues, but introduce their own complexities around how to combine adaptors from multiple tasks.

Most continual learning algorithms assume knowledge of when task boundaries occur: when the model should transition from learning task $k$ to learning task $k+1$. In practice, data distributions often shift gradually and without clear demarcation. A language model deployed in production sees a continuously evolving distribution rather than clean sequential tasks. Detecting when a significant enough distribution shift has occurred to warrant a new "task" is itself a non-trivial problem, related to the concept drift literature in machine learning.

When task boundaries are not known, many continual learning methods cannot operate correctly. Methods that store one set of reference parameters per task need to know when to create a new task reference. Methods that use task-specific output heads need to know when to add a new head. Methods that compute gradient constraints need task labels to select which past examples to include in the constraint set. All of these require some form of task boundary detection or change-point detection as a prerequisite.

Developing continual learning methods that handle unknown task boundaries, sometimes called online continual learning or boundary-free continual learning, is an active research area. These methods must use statistical signals from the data itself, such as sudden increases in validation loss, changes in the data distribution detected by auxiliary models, or drops in prediction confidence, to infer when the distribution has shifted.

Forgetting does not operate uniformly across all types of knowledge. Highly task-specific knowledge (specific vocabulary of a niche domain, formatting conventions of a particular document type) is more vulnerable to forgetting than general, heavily reinforced knowledge (basic grammar patterns that appear across all tasks, fundamental reasoning patterns that are used in many contexts). This non-uniform vulnerability means that monitoring aggregate metrics like BWT or average accuracy can mask important qualitative changes in what the model can and cannot do.

A language model might maintain high average accuracy on downstream benchmarks while losing subtle stylistic capabilities or domain-specific knowledge that only appears in narrow test cases. A medical language model that retains 95% average accuracy after continual fine-tuning might still fail on a specific class of medication dosage questions that appeared rarely in the historical training data. Aggregate metrics would not detect this. Task-specific diagnostic evaluations are necessary to catch domain-specific forgetting that aggregate metrics miss.

A final consideration that the formal problem statement obscures is the value of forward transfer alongside the need to avoid backward forgetting. An ideal continual learning system would not just avoid forgetting; it would use its growing history of tasks to become better at new tasks, accumulating general knowledge that transfers broadly. Systems that achieve near-zero BWT through extreme parameter isolation may fail to exhibit this positive forward transfer, because they prevent any cross-task influence, including helpful influence.

The best continual learning methods achieve a balance: they prevent the harmful backward interference that causes forgetting while preserving the helpful forward transfer that makes sequential learning more efficient than learning each task in isolation. This balance is the deepest design challenge in the field, and it is not yet solved in general. Every method makes different trade-offs, and the right balance depends on the specific application, the similarity between tasks, and the resource constraints available.

Continual learning addresses the challenge of training models on sequences of tasks without forgetting previously learned knowledge. The central obstacle is catastrophic forgetting: when a neural network's parameters are updated to minimize loss on a new task, the updates overwrite the weights that encoded knowledge from previous tasks.

The key concepts from this chapter are:

• Plasticity-stability dilemma: the fundamental trade-off between learning new information quickly (plasticity) and retaining old information accurately (stability). Every continual learning method makes a choice about where to sit on this spectrum.
• Catastrophic forgetting: the sharp, near-total loss of performance on earlier tasks following sequential gradient updates on new tasks. It is caused by the optimizer following the gradient of the new task loss without any awareness of or constraint from old tasks.
• Loss landscape perspective: forgetting occurs because the optimizer moves the parameters from one task's loss basin into another. The two basins are typically separated in high-dimensional parameter space, and unconstrained gradient descent does not find configurations that satisfy both simultaneously.
• Backward Transfer (BWT): the change in performance on earlier tasks caused by training on later tasks. Negative values indicate forgetting; zero means no degradation.
• Forward Transfer (FWT): the change in performance on new tasks attributable to prior task training. Positive values indicate beneficial transfer from shared representations.
• Task-Incremental Learning (Task-IL): the easiest scenario, where task identity is provided at test time. The model can route inputs to task-specific output heads, limiting the forgetting problem to the shared backbone.
• Domain-Incremental Learning (Domain-IL): a medium-difficulty scenario, where the task type is fixed but input distributions shift and no task identity is provided at test time. The shared output head means all domains must be handled by a single classifier.
• Class-Incremental Learning (Class-IL): the hardest scenario, where new classes are added over time and the model must classify across all classes without task identity. Recency bias and growing output space make this particularly challenging.
• Gradient interference: the optimization-level explanation for forgetting, where gradients for new tasks point in directions that conflict with the gradient constraints needed to maintain old task performance.
• Representation drift: the phenomenon where internal representations shift during sequential training, potentially degrading downstream components even when top-level task accuracy is maintained.

The upcoming chapters in this part of the book address the solutions. Regularization-based methods add penalty terms to the loss to protect important parameters from large updates. Replay-based methods maintain a memory of past examples and interleave them with new data. Parameter isolation methods allocate separate model capacity to different tasks to prevent interference. And meta-learning approaches train models specifically to learn new tasks without forgetting, rather than addressing forgetting as an afterthought.

Each approach embodies a different answer to the same question: what resource should we spend to preserve the past?

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about continual learning, catastrophic forgetting, and learning scenarios.