Continual Learning Evaluation: BWT, FWT, and Benchmarks

When a model learns a new task, how do you know whether it helped or hurt? That question turns out to be surprisingly difficult to answer. Standard machine learning evaluation compares a model's accuracy on a held-out test set after training is complete. Continual learning evaluation must do something harder: track how well a model retains what it knew before, measure what it gains from new experience, and assess whether the two goals work against each other.

The difficulty is not merely computational. It is conceptual. Evaluating a continual learner requires you to reason simultaneously about three axes that standard evaluation collapses into one. First, there is the question of retention: does the model still know what it used to know? Second, there is the question of acquisition: does the model learn new tasks as well as it would if it were trained on them in isolation? Third, there is the question of transfer: does experience with earlier tasks make later learning easier, or does it get in the way? A single accuracy number on the most recent task reveals nothing about the first two axes and conflates the second and third.

This chapter covers the metrics and protocols that make continual learning measurable. You will see why overall accuracy alone is insufficient, how forward transfer and backward transfer capture the memory-plasticity tradeoff at the heart of continual learning, and which benchmark suites the research community uses to compare methods. Together, these evaluation tools let you distinguish a model that adapts from one that simply forgets. By the end, you will have a complete quantitative vocabulary for characterizing any continual learning system, and a clear sense of where each benchmark and metric is most informative.

In a single-task setting, the evaluation ritual is simple: train the model, measure accuracy on a held-out test set drawn from the same distribution as the training data, and report that number. The score is a complete summary of model quality because the model only ever needs to handle one task. The test set is fixed, the training data is fixed, and the only question is how well the model generalizes from one to the other.

Continual learning trains the same model sequentially on tasks $T_1, T_2, \ldots, T_N$. After training on task $T_k$, a standard evaluation would report accuracy on a test set for $T_k$. This misses two critical questions:

• Has performance on earlier tasks $T_1, \ldots, T_{k-1}$ degraded?
• Did learning $T_{k-1}$ help or hinder learning $T_k$?

A model could score perfectly on $T_k$ while having catastrophically forgotten everything about $T_1$. That same model might also be harder to train on $T_k$ because the weight updates from $T_{k-1}$ moved parameters into a poorly suited region. Standard test accuracy captures neither of these dynamics.

Consider what happens when you fine-tune a large language model on a legal document classification task and then fine-tune it again on a medical summarization task. After the second fine-tuning, you might report high accuracy on medical summarization and declare success. But if you go back and test the model on legal classification, you may find the performance has collapsed. More subtly, you might ask whether the model learned medical summarization better or worse than a fresh model would have, given the pre-existing legal representations in its weights. Standard evaluation asks only the first question and ignores the second and third entirely.

The evaluation framework for continual learning therefore needs metrics that look backward at what was retained, look forward at what was transferred, and assess the full matrix of task performance over the entire learning sequence. Building this framework starts with a single data structure: the accuracy matrix.

The foundation of continual learning evaluation is a matrix of test accuracies recorded throughout training. After training on each task $T_i$, you evaluate the model on every task $T_j$ seen so far. This produces an accuracy matrix $R$, where entry $R_{i,j}$ is the accuracy on task $T_j$ immediately after training on task $T_i$.

To understand what this matrix captures, consider a five-task example. Immediately after finishing task $T_1$, you record a single row: the model's accuracy on $T_1$ itself. After finishing $T_2$, you record two values: accuracy on $T_2$ (the new task) and accuracy on $T_1$ (which may have degraded). The matrix grows one row at a time:

where:

• $R_{i,j}$ is the test accuracy on task $T_j$ after the model has been trained sequentially through task $T_i$
• $R_{i,i}$ (the diagonal) captures immediate performance right after the model finishes learning task $T_i$
• $R_{i,j}$ for $j < i$ (lower-left entries) captures how well the model retains task $T_j$ after subsequent training on tasks $T_{j+1}, \ldots, T_i$
• $N$ is the total number of tasks

The diagonal entries represent the model at its best for each task in the sequential setting. The off-diagonal entries track degradation. A matrix with high diagonal values and off-diagonal values close to those diagonal values indicates excellent retention. A matrix where off-diagonal values collapse toward chance indicates catastrophic forgetting.

Reading the accuracy matrix provides an immediate visual diagnosis. If the matrix looks nearly uniform across every row for a given column, the model has maintained consistent performance on that task throughout training. If the entries in a column drop sharply from left to right (that is, from earlier rows to later ones), the model is forgetting that task as it learns new ones. If the diagonal entries are all high but the off-diagonal entries drop, that is the signature of a model that learns each task perfectly in isolation but loses what it once knew. If the diagonal entries themselves are lower than expected, the model has difficulty learning new tasks, perhaps because prior weight adjustments have made certain regions of parameter space harder to reach.

The matrix is the raw material from which every summary metric is derived. Rather than discarding all this information into a single number, the matrix preserves the full history of the model's performance trajectory. When you want to understand why a method works or fails, you return to the matrix and examine which entries change and by how much.

The simplest scalar summary is average accuracy after the model has completed all $N$ tasks. We read the final row of the matrix (after training on all tasks) and average across all task columns:

where:

• $R_{N,j}$ is the accuracy on task $T_j$ after the model has been trained through all $N$ tasks
• $N$ is the total number of tasks

Average accuracy is easy to compute and easy to compare across methods. Its weakness is that it conflates forgetting with never learning. A method that learns every task perfectly but forgets everything old, and a method that never forgets but learns nothing new, can both produce the same average accuracy score if the initial and final performance happen to balance out. More concretely, imagine a 5-task scenario where the naive baseline achieves 95% accuracy on the final task but only 50% on the first four (chance for binary classification). The average is $0.50 \times 4/5 + 0.95 \times 1/5 = 0.59$. Now compare with a method that achieves 70% on every task. The second method's average is also $0.70$, which looks better on paper, but the underlying dynamics are completely different. One model forgets catastrophically; the other learns modestly but retains everything.

Despite this weakness, average accuracy remains the most widely reported continual learning metric because it provides a direct comparison to a natural upper bound: a joint model trained on all tasks simultaneously. The joint model sees all training data at once and is not subject to forgetting or interference. The gap between a continual learner's average accuracy and the joint model's accuracy summarizes the total cost of sequential training. A continual learning method is generally considered competitive if it closes most of this gap, even if it does not match the joint model exactly.

A related metric is average accuracy at each training step, which tracks the final average and how average performance evolves as the model absorbs new tasks. This trajectory often reveals a characteristic sawtooth pattern in naive baselines: accuracy jumps when a new task is freshly learned and then falls as subsequent training causes forgetting. Methods that successfully mitigate forgetting show a much smoother, steadily improving trajectory.

Backward transfer (BWT) measures how much learning new tasks affects performance on previously learned tasks. The question it answers is: after the entire learning sequence is complete, how does each task's final accuracy compare to the accuracy it had right after being trained?

Formally, we compare the final row of the accuracy matrix against the diagonal:

where:

• $R_{N,j}$ is the accuracy on task $T_j$ after the model has finished training on all $N$ tasks
• $R_{j,j}$ is the accuracy on task $T_j$ right after the model finished training on it (the diagonal entry, representing peak accuracy for that task in the sequential setting)
• $N$ is the total number of tasks

The numerator for each task, $R_{N,j} - R_{j,j}$, is the change in accuracy from when the model was freshly trained on that task to when all subsequent training is complete. Negative values mean the model lost accuracy over time; positive values mean the model somehow improved on a task after no longer training on it.

Backward transfer is typically negative. A value of $-0.20$ means that, on average, tasks lose 20 percentage points of accuracy after the model learns subsequent tasks. A value near zero indicates strong retention. A positive value would mean that subsequent tasks improved performance on earlier ones, which occasionally happens through beneficial weight sharing when tasks are closely related, such as when two tasks both require detecting low-level visual features like edges or textures.

The key insight backward transfer provides is that it isolates the retention problem from the learning problem. You can have excellent average accuracy with very poor BWT if the model happens to learn later tasks well enough to compensate for forgetting early ones. Separating these effects reveals the true character of a continual learning method. A method that achieves high average accuracy through strong BWT is doing something fundamentally different from one that achieves the same average accuracy through high initial performance on the current task at the expense of everything prior.

In practice, BWT values tend to cluster in a few interpretable ranges. A BWT close to zero (within a few percentage points) indicates that the method handles forgetting well. Approaches like parameter regularization (EWC, SI) and replay-based methods typically achieve BWT values between $-0.05$ and $-0.15$ on standard benchmarks, while still accommodating new tasks.

A strongly negative BWT (below $-0.30$) signals catastrophic forgetting. A naive fine-tuning baseline on permuted MNIST often achieves BWT around $-0.70$ or worse: the model's performance on early permutations collapses almost entirely as it adapts to later permutations. On Split CIFAR-100, where tasks involve distinguishing more semantically distinct categories, naive fine-tuning BWT can be even more extreme because there is less incidental weight overlap between tasks.

Positive BWT is rare but real. It occurs when tasks share structure that reinforces each other, for instance when both tasks require detecting edges or recognizing phonemes. When you observe positive BWT, it suggests that the tasks in the benchmark are more related than typical continual learning scenarios assume, which may limit the benchmark's ability to stress-test forgetting. Researchers who observe positive BWT should look carefully at whether their benchmark requires the model to update weights in conflicting directions across tasks, or whether the tasks are so similar that learning one simply refines representations useful for the other.

One subtlety in interpreting BWT is the choice of reference point. Standard BWT compares final accuracy against the diagonal of the accuracy matrix, which is the accuracy right after training on each task in the sequential setting. This is different from comparing against the isolated baseline $b_j$. A method could achieve BWT near zero by learning each task poorly initially (low diagonal) but maintaining that low performance consistently. This would look like good retention even though the model never learned the tasks well. For this reason, BWT should always be reported alongside the diagonal values themselves or alongside average accuracy, not in isolation.

Forward transfer (FWT) measures how much learning earlier tasks helps the model learn future tasks more quickly or accurately. The standard definition compares each task's accuracy right after training against the baseline accuracy achieved when training that task from scratch in isolation.

One common formulation computes:

where:

• $R_{j,j}$ is the accuracy on task $T_j$ right after the model finishes training on it, having already been trained on tasks $T_1, \ldots, T_{j-1}$
• $b_j$ is the accuracy on task $T_j$ when a fresh model is trained on it from scratch in isolation
• $N$ is the total number of tasks
• The sum starts at $j = 2$ because there is no prior task experience before the first task

A positive FWT means that earlier tasks improved the model's ability to learn later tasks. A negative FWT means the opposite: earlier training made it harder to learn new tasks, a phenomenon sometimes called negative transfer or task interference. Negative transfer is often caused by catastrophic interference in the opposite direction from forgetting: the model does not just lose what it knew about old tasks, it also develops features that actively conflict with what new tasks need.

Forward transfer is particularly important in the meta-learning interpretation of continual learning, where the goal is to avoid forgetting and become a better learner over time. A system with high positive FWT is building increasingly general representations that transfer to new domains with less data, which is a hallmark of human-like generalization. Humans do more than avoid forgetting how to ride a bicycle when they learn to drive a car. They also find that prior physical coordination experience, developed through riding, makes driving easier to learn. The goal of continual learning research is to build systems that exhibit this same cumulative benefit from accumulated experience.

In practice, achieving positive FWT with neural networks is difficult. Standard gradient-based training does not naturally build representations that generalize across unrelated tasks. FWT tends to be positive only when tasks are semantically or structurally related, when the model has a sufficiently large capacity to develop shared representations, or when the training procedure explicitly encourages transfer through regularization or architectural choices.

BWT and FWT together characterize the memory-plasticity tradeoff in a single pair of numbers. Methods that aggressively protect old knowledge (high BWT, meaning close to zero) often sacrifice plasticity: they change weights so conservatively that new tasks cannot be learned as effectively, leading to reduced FWT or even negative FWT. Methods that prioritize rapid adaptation to new tasks (high FWT) often allow weights to drift, causing forgetting (low BWT, meaning strongly negative).

This tradeoff is not always forced. Replay-based methods can achieve both high BWT and positive FWT by reusing stored representations that are helpful for new tasks. When the model rehearses old examples while learning a new task, it maintains the features needed for old tasks while simultaneously exposing the new task's optimizer to those features, which can help if the new task is related. Understanding where a method falls in the BWT-FWT space reveals its practical strengths and tells you whether it is a better fit for scenarios that require long-term retention or scenarios that require rapid adaptation.

The BWT-FWT plane is a useful conceptual tool for comparing methods. Methods in the upper-right quadrant (high BWT, positive FWT) are ideal. Methods in the lower-right quadrant (high BWT, negative FWT) preserve old knowledge but interfere with new learning, a pattern typical of overly aggressive regularization. Methods in the upper-left quadrant (low BWT, positive FWT) adapt rapidly but forget, which matches naive fine-tuning. Methods in the lower-left quadrant fail on both axes simultaneously. Most realistic methods occupy the territory between these extremes, and the goal of continual learning research is to develop methods that reliably push toward the upper-right corner of this space.

The forgetting measure $F$ is closely related to BWT but focuses on the worst-case drop from peak to final performance, rather than comparing against the diagonal entry immediately after training. For each task, it finds the highest accuracy ever achieved on that task across all evaluation points during the training sequence, and measures how far the final accuracy has fallen from that peak.

The distinction matters because a task might continue to improve slightly after its initial training, perhaps because representations from subsequent tasks reinforce it. In that case, the diagonal entry $R_{j,j}$ would understate the task's actual best performance, and BWT would give a misleadingly optimistic picture. The forgetting measure avoids this by anchoring to the true maximum.

The per-task forgetting for task $T_j$ is:

where $\max_{k \in \{j, \ldots, N-1\}} R_{k,j}$ is the highest accuracy achieved on task $T_j$ at any point after the model was first trained on it, and $R_{N,j}$ is the final accuracy after all training is complete.

The aggregate forgetting measure averages over all tasks except the last (which has no subsequent training to cause forgetting):

Forgetting is always non-negative by construction, since it compares the final accuracy against the maximum ever achieved. A value of zero means the model never fell below its peak on any task. High values indicate severe degradation relative to what the model once knew.

The difference between $F$ and BWT is subtle but meaningful. BWT compares final accuracy against the accuracy right after training each task: if a task is learned poorly initially but then somehow improves through beneficial transfer from later tasks (positive BWT), BWT can still be non-negative even if later degradation occurs. The forgetting measure captures the degradation more directly by anchoring to the highest point ever reached. This makes $F$ a more conservative metric that is less susceptible to being inflated by cases where initial learning is poor but later transfer happens to help.

In practical reporting, both metrics convey useful information. BWT tells you how the model's final state compares to its state right after first learning each task, which is the most natural comparison point if the goal is to preserve knowledge from when it was freshest. The forgetting measure tells you how much of the model's all-time best performance on each task was ultimately lost, regardless of when that peak occurred. For tasks where knowledge accumulates gradually through repeated exposure, the forgetting measure is the more appropriate choice.

Intransigence measures the model's resistance to learning new tasks due to the inertia accumulated from prior training. Where forgetting asks "how much did the model lose from what it knew?", intransigence asks "how hard was it to learn new tasks given what the model already knew?". These are two distinct failure modes that must be measured separately.

The per-task intransigence for task $T_j$ is:

The aggregate intransigence averages over all tasks:

where:

• $b_j$ is the accuracy on task $T_j$ when trained from scratch in isolation (the best achievable performance without interference)
• $R_{j,j}$ is the accuracy on task $T_j$ when trained sequentially after tasks $T_1, \ldots, T_{j-1}$ (the model's result in the sequential setting)
• A positive value of $I_j$ means prior training hurt learning of task $T_j$
• A negative value of $I_j$ (though rare) would mean prior training helped

The relationship between intransigence and forward transfer is direct: intransigence is essentially the negative of FWT (with the average including the first task as well as subsequent ones). High intransigence corresponds to negative FWT: prior experience is making new learning harder, not easier. Low or negative intransigence corresponds to positive FWT: prior experience is a net benefit.

Intransigence is especially relevant when evaluating regularization-heavy methods. A method that penalizes all weight changes to protect prior tasks will show low forgetting but high intransigence: it cannot update weights freely enough to learn new tasks, so new-task accuracy stays well below what an unconstrained model would achieve. This is the fundamental tension in regularization-based approaches. Elastic Weight Consolidation (EWC), for example, adds a penalty term that slows down changes to weights deemed important for prior tasks. If the penalty coefficient is set too high, the model learns new tasks poorly, producing high intransigence. If set too low, forgetting is insufficiently controlled. Finding the right balance is one of the central practical challenges in deploying regularization-based continual learning.

High intransigence means the model's prior training makes it harder to absorb new information. This is the flip side of forgetting: forgetting measures damage to old knowledge while intransigence measures difficulty acquiring new knowledge. Both are failure modes in continual learning, and both must be monitored to understand a method's overall behavior. A complete evaluation should always report both, since a method that eliminates forgetting by making new learning impossible is not a useful solution.

There is a deeper conceptual reason why intransigence and forgetting both exist and why reducing one often exacerbates the other. The weights of a neural network must simultaneously encode representations for all tasks. When the representation space required by task $j+1$ overlaps significantly with the space required by tasks $1$ through $j$, there is room for the network to accommodate both without conflict, and intransigence is low. When the spaces conflict, any attempt to shift representations toward the new task will partially undo the old ones, and any attempt to freeze the old representations will block the new task from achieving its optimum. This geometric view of the plasticity-stability tradeoff explains why the problem is fundamentally hard and why the metrics of forgetting and intransigence together provide a more complete picture than either one alone.

The metrics above are only meaningful when applied to standard benchmark protocols. The research community has developed several evaluation suites that vary in difficulty, task structure, and the type of forgetting they elicit. Choosing the right benchmark for a given research question is as important as choosing the right metric.

Permuted MNIST is one of the oldest continual learning benchmarks, introduced alongside early work on preventing catastrophic forgetting in neural networks. Each task is a 10-class image classification problem on handwritten digits, but with a different fixed random permutation applied to the pixel positions. The underlying images are from MNIST, the classic dataset of 28x28 grayscale digit images. Because all tasks share the same output space (10 digit classes), this is a domain-incremental setting: the model cannot tell which task it is on from the output labels alone, since every task produces outputs labeled 0 through 9.

Permuted MNIST is relatively easy for modern methods because all tasks have the same input dimensionality and the same number of classes. The permutations differ enough that weights learned for one permutation are not directly reusable for another, creating forgetting pressure. A representation that recognizes the digit "3" in the original orientation must be completely relearned for a permuted version where the pixels of "3" are shuffled into an apparently random pattern. However, the benchmark's weakness is that it does not test semantic transfer: there is no meaningful reason why learning permutation A should help with permutation B, since both permutations look like random noise to any semantically aware representation.

A typical evaluation suite uses 10 to 20 permutations. A naive fine-tuning baseline achieves final average accuracy around 19% on 10 tasks (just above chance), while strong methods achieve over 90%. The benchmark is particularly useful for studying the basic mechanics of forgetting because the task structure is controlled and repeatable. However, because the tasks are all variations on the same digit dataset with no semantic relationship, results on permuted MNIST do not necessarily predict performance on more realistic, semantically diverse task sequences.

Split MNIST divides the 10 MNIST digit classes into 5 binary classification tasks, presented sequentially:

• Task 1: digits 0 vs 1
• Task 2: digits 2 vs 3
• Task 3: digits 4 vs 5
• Task 4: digits 6 vs 7
• Task 5: digits 8 vs 9

This is a class-incremental setting by construction. The difficulty is that the output space changes with each task: early tasks use 2-class outputs, and the model must eventually discriminate all 10 classes without knowing the task identity at test time. Naive fine-tuning achieves roughly 19% final accuracy here as well, since the model forgets early tasks almost entirely by the time it has been trained on all five.

Split MNIST is a strict benchmark for task-agnostic evaluation because the model must disambiguate which task a test example belongs to without a task label. A test image of the digit "0" could belong to task 1, where the model should classify it as class 0 of that binary pair. But in class-incremental evaluation, the model must output the correct class across all 10 possible outputs simultaneously, which means it must implicitly solve a task detection problem on top of the classification problem. Methods that can look up the task identity at test time (task-incremental setting) perform much better, often above 95%, but this is generally considered a weaker and less realistic evaluation condition.

The key pedagogical value of Split MNIST is its clarity. Because the tasks are defined by explicit class splits, you can reason directly about what knowledge is being lost and what the model needs to retain. A model that forgets task 1 while learning task 5 is losing the ability to distinguish 0 from 1, which is interpretable and easy to debug. This interpretability makes Split MNIST a useful first benchmark when developing a new continual learning method.

These benchmarks extend the split approach to more realistic image recognition tasks, introducing significantly more difficulty through greater visual complexity and more fine-grained categories. Split CIFAR-10 divides 10 natural image classes (airplanes, automobiles, birds, cats, deer, dogs, frogs, horses, ships, trucks) into 5 binary tasks. Split CIFAR-100 creates 20 tasks of 5 classes each from the 100-class CIFAR-100 dataset, which spans 20 superclasses including aquatic mammals, fish, flowers, food containers, fruit and vegetables, household electrical devices, and many more.

CIFAR-100 is the more demanding benchmark because the features required to distinguish 100 fine-grained categories are more diverse than MNIST digits. The difference between a beaver and an otter requires detecting subtle texture and shape differences that are far more complex than the stroke patterns distinguishing handwritten digits. The number of tasks is also larger (20 compared to 5 for MNIST), giving more opportunity for forgetting to accumulate over the training sequence.

Methods are typically evaluated in both task-incremental mode (task identity known at test time) and class-incremental mode (task identity unknown). The gap between these two modes reveals how much a method relies on task identity to route examples to the right classifier head. A method with a large task-IL to class-IL performance gap is essentially cheating in the easier setting: it builds a separate classifier per task and then uses the task label to select the right one at test time. This is a valid engineering solution in some applications, but it does not demonstrate continual learning of unified representations.

One important practical note about CIFAR benchmarks is that they require a feature extractor capable of handling natural images. Many papers use a pretrained ResNet backbone, which already has strong visual representations. This creates an implicit confound: the model's ability to handle multiple tasks may partly reflect the richness of the pretrained features rather than the effectiveness of the continual learning algorithm. When reporting results on CIFAR benchmarks, it is important to specify whether the backbone is pretrained and frozen, pretrained and fine-tuned, or trained from scratch, since this choice dramatically affects the absolute performance numbers and the relative advantage of different continual learning strategies.

Tiny ImageNet is a downsampled version of ImageNet with 200 classes at 64x64 resolution, selected from the original 1,000 ImageNet classes to span a diverse range of objects, animals, and scenes. Continual learning benchmarks typically split it into 10 to 40 tasks of 20 to 5 classes each. The larger number of tasks and higher visual complexity make it a considerably harder evaluation than CIFAR benchmarks.

It provides a harder visual recognition challenge than CIFAR-100 because the classes are more diverse and the intra-class variation is higher. Recognizing 200 object categories across wildly different visual domains, from fish and frogs to street furniture and sports equipment, requires representations that span a much broader feature space than the more semantically coherent CIFAR-100 categories.

Tiny ImageNet benchmarks are used to evaluate whether continual learning methods scale to more realistic inputs. Some methods that work well on MNIST and CIFAR fail to generalize to Tiny ImageNet because their memory budgets, replay strategies, or regularization penalties do not scale proportionally with task complexity. A method that stores 20 examples per class to enable replay might work well with 10 MNIST classes but require 200 stored examples per class for Tiny ImageNet to achieve the same quality of replay, and the storage cost of 200 examples times 200 classes at 64x64 resolution quickly becomes prohibitive.

Evaluating on Tiny ImageNet is therefore a useful check on scalability. A continual learning method that performs strongly on MNIST but weakly on Tiny ImageNet is likely overfitting its hyperparameters and design choices to the simpler regime. Methods that maintain their relative advantage across both scales provide stronger evidence that the advantage scales.

CORe50 is designed specifically for object recognition under domain shift, and it differs fundamentally from the class-split benchmarks in the nature of the challenge it poses. It contains 50 household objects, each belonging to one of 10 categories (plug adapters, mobile phones, scissors, light bulbs, cans, glasses, balls, markers, cups, remote controls), captured across 11 recording sessions with varying backgrounds, lighting, and viewpoints.

Unlike the class-split benchmarks where the objects and categories change from task to task, CORe50 presents distributional shift: the same objects look different in different recording sessions because of environmental variation. This is a much more realistic model of the challenges a deployed vision system would face. A robot that learns to recognize a coffee mug in a bright kitchen might be tested on recognizing the same mug in a dim hallway. The object is the same; the appearance is different.

The benchmark supports multiple evaluation protocols. In the new instances (NI) protocol, each session is a task and the object categories remain the same but the visual distribution shifts. In the new categories (NC) protocol, each task introduces new object categories and the visual distribution also changes. In the new instances and categories (NIC) protocol, both the distribution and the categories shift simultaneously, creating the most realistic and demanding evaluation scenario.

CORe50 is particularly relevant for robotics and mobile vision applications where the deployment environment changes continuously. It tests whether continual learning methods handle distributional shift, not just class-incremental splits. A method that excels at class-incremental benchmarks by building a separate classifier per class may fail entirely on CORe50's NI protocol, where the task is always recognizing the same 10 categories but under different visual conditions, because the per-class classifier approach provides no benefit when the categories do not change.

As continual learning has expanded to language models, benchmarks have emerged that sequence NLP classification tasks rather than image tasks. The most common protocols use subsets of GLUE or SuperGLUE benchmarks arranged sequentially. A model might train on sentiment analysis from one dataset, then natural language inference from another, then question answering from a third, then named entity recognition from a fourth. Each task tests different linguistic capabilities and requires different output formats and feature specializations.

Domain-specific classification sequences present a different angle on NLP continual learning. A model might learn to classify medical literature, then legal documents, then financial filings, then social media posts, all with the same text-classification output format but across wildly different vocabularies, writing styles, and domain knowledge requirements. Forgetting in this setting means losing the ability to correctly classify medical texts after adapting to financial language, even though both tasks technically use the same output labels.

Multilingual tasks presented one language at a time form yet another variant. A multilingual model might learn English, then French, then German, then Mandarin. Because languages share grammatical structures and some lexical overlap, transfer can be measured across this sequence. The question of whether early language learning transfers to later languages, and whether later language learning degrades earlier language capability, is practically important for building language models that serve diverse global users.

These NLP benchmarks reveal that large pretrained language models exhibit surprisingly strong resistance to forgetting compared to models trained from scratch. The representations learned during pretraining on massive text corpora are broad enough that fine-tuning on one task rarely completely overwrites them. Catastrophic forgetting in large language models is real but typically less severe than in smaller models trained from random initialization. This phenomenon, often called stability-plasticity in the LLM context, reflects the fact that pretrained representations resist the specific gradient updates produced by individual fine-tuning datasets. The upcoming chapters on continual learning in foundation models will examine this phenomenon in detail, including the conditions under which even large models exhibit significant forgetting and the techniques developed to address it.

Beyond the specific benchmarks, the research community distinguishes several evaluation protocols that affect what continual learning methods can and cannot do during inference. The choice of protocol is as consequential as the choice of benchmark, and comparing methods evaluated under different protocols is typically invalid.

In task-incremental learning (task-IL), the model knows the task identity $t$ at both training and test time. This allows the model to use a separate output head for each task, routing test examples to the right classifier automatically. Because the model never needs to distinguish between tasks, the problem reduces to ensuring that learning task $t+1$ does not damage the features needed by the classifier for task $t$.

Task-IL is the most favorable setting for continual learning methods. It simplifies the problem considerably because the output ambiguity that makes class-IL difficult is removed entirely. The model knows which task is being evaluated and can apply the corresponding output head directly. Many methods that fail in class-IL settings perform well in task-IL because the only challenge remaining is weight interference in shared feature layers, not output disambiguation.

Task-IL is unrealistically easy in many practical scenarios because knowing the task identity at test time is not always available. In a real deployment, the system may not receive explicit signals about which task a given input belongs to. A question-answering system deployed in production does not know whether an incoming query belongs to the "factual lookup" task it was trained on first or the "reasoning" task it was trained on second. For this reason, task-IL results serve as an important upper bound on what can be achieved with a given architecture and training procedure, but they should not be reported as evidence of practical continual learning performance without clearly noting that task identity is provided at test time.

In class-incremental learning (class-IL), the model knows the task identity during training but not at test time. The model must output the correct class from the union of all classes seen so far, without any hint about which task a test example belongs to.

This is significantly harder than task-IL. Without the task identifier, the model must solve an implicit task discrimination problem on top of the classification problem. If the model has learned to classify digits 0 through 1 in task 1 and digits 2 through 3 in task 2, a class-IL test requires the model to correctly identify the digit from a pool of all four classes (0, 1, 2, 3) without knowing whether the image belongs to task 1 or task 2. Naive fine-tuning methods fail badly here because the latest task's output head dominates all predictions. Methods must either store explicit examples for replay, build task-agnostic representations, or employ architectural tricks to perform well.

Class-IL is the standard setting for most published continual learning comparisons because it represents a realistic scenario: a deployed model receives new data over time and must continue classifying correctly for all classes it has ever seen. A spam filter that learns to recognize a new type of phishing email should continue working correctly on all previously known spam categories as well as on the new category. The inability to provide a "this email might be of type X" hint at test time means the model must handle the entire accumulated output space simultaneously.

One important nuance in class-IL evaluation is how the output head is structured. Some methods grow the output layer as new classes arrive (expanding head), while others fix the output size from the beginning (fixed head). Expanding heads require knowing the total number of eventual classes in advance or performing dynamic architecture modification during training. Fixed heads require the model to partition a fixed output space across tasks. These architectural choices interact significantly with the continual learning method and must be specified when reporting class-IL results.

In domain-incremental learning (domain-IL), the output space is fixed but the input distribution shifts. For example, a sentiment classifier trained on movie reviews must then learn to classify restaurant reviews, then tech reviews, using the same positive/negative output labels throughout. The model does not know which domain a test example comes from.

Domain-IL is relevant for deployed systems that encounter distribution shift over time without explicitly adding new output categories. It tests whether the model builds domain-invariant features or whether its features become too specialized to the training distribution. A model that is highly specialized to the vocabulary and writing style of movie reviews may perform poorly on restaurant reviews even though the underlying task, distinguishing positive from negative sentiment, is the same.

The CORe50 NI protocol is a visual example of domain-IL: the same object categories exist throughout, but the recording sessions (domains) change. The challenge is entirely distributional, not categorical. Domain-IL is generally easier than class-IL for methods that work by expanding output capacity, since no new output heads are needed. However, it is harder for methods that rely on detecting shifts in the output distribution to trigger replay or regularization, since the output distribution is identical across domains by construction.

One underappreciated aspect of evaluation protocol is how often the accuracy matrix is measured. Some papers evaluate only at the end of training on each task (coarse-grained). Others evaluate after every gradient update (fine-grained). The choice affects what you can infer about forgetting dynamics and makes a substantial difference in the cost of running the evaluation.

Fine-grained evaluation reveals whether forgetting is gradual or catastrophic, whether it happens mostly at the beginning of a new task's training or continues throughout, and whether certain training checkpoints are particularly vulnerable. A model might lose most of its performance on task 1 during the first hundred gradient steps of task 2 training, and then maintain a stable (if degraded) level for the remaining training steps. This would appear as a single large drop in coarse-grained evaluation. Fine-grained evaluation would reveal the dynamics of that drop and potentially identify an early stopping point that balances new-task learning against old-task retention.

Coarse-grained evaluation is sufficient for comparing final performance but misses these dynamics entirely. For benchmarking purposes, end-of-task evaluation is standard because it reduces the computational cost and simplifies reporting. However, for debugging a new method, fine-grained tracking of the accuracy matrix is far more informative. Observing where in training each task's performance collapses, and how quickly, can reveal whether the problem is a learning rate that is too high, a regularization coefficient that is insufficiently calibrated, or a replay strategy that does not sample old examples frequently enough.

The evaluation frequency also affects the values of BWT and the forgetting measure. With coarse-grained evaluation, you only have access to the diagonal entries of the accuracy matrix and the final row. With fine-grained evaluation, you have access to every row, which gives you the full trajectory of each task's accuracy and allows the forgetting measure to anchor at the true peak rather than just at the end of each task's initial training phase.

This section builds a framework to evaluate a continual learning scenario from scratch, computing the accuracy matrix and all derived metrics.

We will simulate a simple continual learning run on Split MNIST: 5 sequential binary classification tasks, each distinguishing a different pair of digit classes. The "model" is represented by a simulated accuracy matrix that reproduces the forgetting pattern of a naive gradient descent baseline and a replay-based method. Using simulation rather than a full MNIST training loop keeps the focus on the evaluation machinery rather than the training mechanics, while still producing realistic-looking metric values.

The replay method achieves higher average accuracy. But average accuracy alone does not tell us whether the gap comes from better retention of old tasks, faster learning of new tasks, or both. That is why we need BWT and FWT. The numerical difference between methods on average accuracy often understates the true behavioral difference, because some of the naive model's forgetting is partially offset by the fact that its most recently trained task performs well.

The naive method shows strongly negative BWT, confirming catastrophic forgetting. The replay method's BWT is much closer to zero, showing that replaying past examples substantially preserves old task performance. The magnitude of the BWT difference between the two methods is much larger than the average accuracy difference, which illustrates exactly why BWT is a more diagnostic metric than average accuracy for understanding forgetting behavior.

Both methods show slightly negative FWT in this simulation. This is typical for Split MNIST: the tasks are designed to be distinct enough that prior training does not transfer positively. The replay method's FWT is closer to zero because replaying diverse examples helps maintain useful general features that modestly benefit new task learning, even when the tasks themselves are not semantically related.

Forgetting confirms the large retention gap between the two methods. Intransigence is near zero for both, which is expected: naive fine-tuning suffers from forgetting but not from rigidity, and replay can still learn new tasks freely because it does not impose weight change penalties. This is a clean example of the asymmetry between forgetting and intransigence: a method can have severe forgetting (overwriting old knowledge) without having any intransigence (blocking new learning), and vice versa.

The trajectory plots tell a story that the summary metrics cannot fully convey. For naive fine-tuning, each task's accuracy is highest at the moment of its own training, then falls precipitously as subsequent tasks overwrite the shared weights. The drop is not gradual. Task 1's accuracy falls most steeply because it is followed by four additional tasks, each contributing further degradation. Task 4's accuracy falls less because only one task follows. The shape of the trajectories reflects the geometric decay built into the simulation, but real networks show similar patterns.

For the replay method, the trajectories are almost horizontal. The periodic rehearsal of stored examples continuously refreshes the model's memory of earlier tasks, preventing the cascade of forgetting visible in the naive baseline. The residual downward slope visible in some task trajectories for the replay method reflects the imperfect coverage of past distributions by a finite replay buffer.

The key parameters governing this evaluation framework are:

• N: Number of tasks. Longer sequences stress-test forgetting more severely because each task has more subsequent training to degrade its performance. A method that performs well on 5-task sequences may fail on 20-task sequences if its forgetting mitigation does not scale.
• Evaluation protocol: Task-IL versus class-IL versus domain-IL determines what task information the model can use during inference, fundamentally changing the difficulty of the problem and the ranking of methods.
• Baseline $b_j$: How the isolated training baseline is computed affects FWT and intransigence interpretation. The baseline should use the same architecture and training budget as the continual learner, with no other tasks present, to ensure a fair comparison.
• Evaluation frequency: End-of-task evaluation (standard) versus per-update tracking (for dynamics analysis). Fine-grained tracking reveals when during training forgetting occurs and how rapidly it progresses.

The metrics described in this chapter capture different facets of the memory-plasticity tradeoff, but they carry important limitations that practitioners should understand before relying on them to guide research decisions or deployment choices.

Task boundary dependence. BWT, FWT, and the forgetting measure all require clear task boundaries: you need to know when training on one task ends and the next begins, so you can record the diagonal entries of the accuracy matrix. In real-world continual learning, task boundaries are often implicit or unknown. A deployed model receiving a steady stream of data does not know when the distribution has shifted. The shift from one user behavior pattern to another, from one product category to another, or from one linguistic register to another, may be gradual and unlabeled. Evaluation protocols that require explicit boundaries do not directly translate to these scenarios. Researchers have begun developing boundary-free evaluation protocols that measure forgetting through held-out test sets sampled at regular time intervals, but these are less established and not yet standardized across the literature.

The benchmark gap. Permuted MNIST, Split CIFAR, and similar benchmarks have well-known failure modes: they are small, they involve simple classification tasks, and they use fixed task sequences that may not reflect realistic deployment patterns. A method that achieves state-of-the-art BWT on permuted MNIST may fail completely on real-world data streams where tasks are noisy, overlapping, or presented in non-stationary order. Tasks in practice are rarely clean binary or 5-class problems with sharp boundaries. They are more often continuous distributions that shift gradually, with overlapping class spaces and varying data quality. Evaluation on multiple benchmarks with different characteristics, including at least one that tests distributional shift (like CORe50) and one that involves natural language, is necessary to build confidence in a method's real-world performance.

Task identity leakage. Many evaluation protocols claim to test class-IL or domain-IL settings, but implicitly allow the model to infer task identity from context. For instance, if each task uses a different subset of output classes, a model that observes which output logits are active during training can infer the current task at test time. If a model has been trained to classify only classes 0 and 1 during task 1, it may learn to produce near-zero logits for all other classes. At test time, the model can effectively determine it is being queried about task 1 by observing which logits are large. Careful benchmark design and evaluation protocols are needed to prevent this leakage, including scrambling output indices across tasks and using uniform random initialization of output heads at each task transition.

Metric gaming. Each summary metric can be optimized at the expense of others, and researchers who are not careful may inadvertently design methods that improve a reported metric without improving actual continual learning capability. A method could achieve near-zero BWT by using a separate model for each task (progressive neural networks), but this would not scale to many tasks and would have trivial FWT. Average accuracy can be maximized by a method that simply allocates more capacity to the last task seen, at the expense of all earlier tasks, if the benchmark's final evaluation happens to weight recent tasks more. When comparing methods, it is important to report the full set of metrics rather than cherry-picking the one that favors a particular approach. The community has gradually moved toward requiring that papers report at least average accuracy, BWT, and FWT together, though the adoption of forgetting and intransigence as standard reports remains incomplete.

The replay data assumption. Methods that store and replay past examples require that past data remains accessible indefinitely, or at least for the duration of the training sequence. In many practical scenarios, data from earlier tasks is deleted for privacy, storage, or regulatory reasons. A medical AI system trained on patient data from hospital A cannot legally retain that data when it moves to hospital B. Evaluation that includes replay-based methods alongside replay-free methods must make clear what data access assumptions each method requires, otherwise the comparison is unfair. This is especially important because replay methods consistently achieve the best benchmark numbers, and a practitioner who cannot legally retain past data would be misled by a leaderboard that does not distinguish replay-constrained from unconstrained methods.

Sensitivity to task order. The accuracy matrix, and all metrics derived from it, depend on the order in which tasks are presented. A task sequence where the most difficult task comes first will produce different BWT values than a sequence where the same task comes last, because the model has more or fewer subsequent tasks to overwrite its weights. Most benchmarks use a fixed task order, which means results are specific to that order and may not generalize to other orderings. Some papers evaluate methods across multiple random orderings and report mean and variance of metrics. This is a more thorough and honest evaluation approach, but it multiplies the computational cost by the number of orderings evaluated and is not yet universal practice.

The continual learning versus multi-task learning comparison. Average accuracy in continual learning is most naturally compared to the joint model trained on all tasks simultaneously. But the joint model is not always the right reference point. In scenarios where data arrives truly sequentially and storing all past data is impossible, the joint model may not be a feasible baseline. In those cases, the more relevant comparison might be to a model that stores only a small buffer of past examples (a constrained replay baseline) rather than the entire training set. Framing the comparison carefully, and stating explicitly what baselines are achievable in the target deployment scenario, makes evaluation results much more actionable.

Continual learning evaluation measures how well a model retains old knowledge while acquiring new knowledge. The key concepts in this chapter are:

• The accuracy matrix $R$ is the foundation: entry $R_{i,j}$ records performance on task $T_j$ after training through task $T_i$. All summary metrics derive from this matrix, and examining it directly reveals the qualitative character of a method's behavior.
• Average accuracy after all tasks summarizes overall performance but conflates retention and learning. It should always be supplemented with more diagnostic metrics.
• Backward transfer (BWT) measures how much subsequent training degrades earlier task performance. Negative values indicate forgetting; values near zero indicate strong retention. BWT is the most direct measure of catastrophic forgetting.
• Forward transfer (FWT) measures how much earlier tasks help or hinder learning new tasks. Positive values indicate beneficial transfer; negative values indicate task interference. FWT captures whether the model is becoming a better learner over time.
• Forgetting measure captures the worst-case drop from peak to final performance on each task. It is more conservative than BWT and anchors to the true maximum rather than the post-training diagonal.
• Intransigence captures how much prior training makes it harder to absorb new tasks. It is the flip side of forgetting and is particularly important for evaluating regularization-based methods that protect old knowledge at the cost of new learning.
• Standard benchmarks (permuted MNIST, split CIFAR, CORe50, NLP sequences) provide controlled settings for comparing methods. Each benchmark tests different aspects of forgetting and transfer, and evaluating on multiple benchmarks provides stronger evidence than any single one.
• Evaluation protocols (task-IL, class-IL, domain-IL) define what task information is available at test time. Class-IL is the standard and hardest setting. Task-IL results should be clearly labeled as using privileged task identity information.
• Every metric and benchmark has limitations. Honest evaluation reports the full metric suite, specifies the exact evaluation protocol, and acknowledges what assumptions about data access and task boundaries are embedded in the comparison.

With these tools, you can characterize any continual learning method beyond a single accuracy number and understand exactly where it succeeds and fails in the memory-plasticity tradeoff. The metrics are the vocabulary for a precise technical conversation about a continual learning system's behavior, and the benchmarks are the shared language that makes comparisons across the research community meaningful.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about continual learning evaluation.