Continual Learning Evaluation › Continual Learning Benchmarks

The metrics above are only meaningful when applied to standard benchmark protocols. The research community has developed several evaluation suites that vary in difficulty, task structure, and the type of forgetting they elicit. Choosing the right benchmark for a given research question is as important as choosing the right metric.

Permuted MNIST is one of the oldest continual learning benchmarks, introduced alongside early work on preventing catastrophic forgetting in neural networks. Each task is a 10-class image classification problem on handwritten digits, but with a different fixed random permutation applied to the pixel positions. The underlying images are from MNIST, the classic dataset of 28x28 grayscale digit images. Because all tasks share the same output space (10 digit classes), this is a domain-incremental setting: the model cannot tell which task it is on from the output labels alone, since every task produces outputs labeled 0 through 9.

Permuted MNIST is relatively easy for modern methods because all tasks have the same input dimensionality and the same number of classes. The permutations differ enough that weights learned for one permutation are not directly reusable for another, creating forgetting pressure. A representation that recognizes the digit "3" in the original orientation must be completely relearned for a permuted version where the pixels of "3" are shuffled into an apparently random pattern. However, the benchmark's weakness is that it does not test semantic transfer: there is no meaningful reason why learning permutation A should help with permutation B, since both permutations look like random noise to any semantically aware representation.

A typical evaluation suite uses 10 to 20 permutations. A naive fine-tuning baseline achieves final average accuracy around 19% on 10 tasks (just above chance), while strong methods achieve over 90%. The benchmark is particularly useful for studying the basic mechanics of forgetting because the task structure is controlled and repeatable. However, because the tasks are all variations on the same digit dataset with no semantic relationship, results on permuted MNIST do not necessarily predict performance on more realistic, semantically diverse task sequences.

Split MNIST divides the 10 MNIST digit classes into 5 binary classification tasks, presented sequentially:

• Task 1: digits 0 vs 1
• Task 2: digits 2 vs 3
• Task 3: digits 4 vs 5
• Task 4: digits 6 vs 7
• Task 5: digits 8 vs 9

This is a class-incremental setting by construction. The difficulty is that the output space changes with each task: early tasks use 2-class outputs, and the model must eventually discriminate all 10 classes without knowing the task identity at test time. Naive fine-tuning achieves roughly 19% final accuracy here as well, since the model forgets early tasks almost entirely by the time it has been trained on all five.

Split MNIST is a strict benchmark for task-agnostic evaluation because the model must disambiguate which task a test example belongs to without a task label. A test image of the digit "0" could belong to task 1, where the model should classify it as class 0 of that binary pair. But in class-incremental evaluation, the model must output the correct class across all 10 possible outputs simultaneously, which means it must implicitly solve a task detection problem on top of the classification problem. Methods that can look up the task identity at test time (task-incremental setting) perform much better, often above 95%, but this is generally considered a weaker and less realistic evaluation condition.

The key pedagogical value of Split MNIST is its clarity. Because the tasks are defined by explicit class splits, you can reason directly about what knowledge is being lost and what the model needs to retain. A model that forgets task 1 while learning task 5 is losing the ability to distinguish 0 from 1, which is interpretable and easy to debug. This interpretability makes Split MNIST a useful first benchmark when developing a new continual learning method.

These benchmarks extend the split approach to more realistic image recognition tasks, introducing significantly more difficulty through greater visual complexity and more fine-grained categories. Split CIFAR-10 divides 10 natural image classes (airplanes, automobiles, birds, cats, deer, dogs, frogs, horses, ships, trucks) into 5 binary tasks. Split CIFAR-100 creates 20 tasks of 5 classes each from the 100-class CIFAR-100 dataset, which spans 20 superclasses including aquatic mammals, fish, flowers, food containers, fruit and vegetables, household electrical devices, and many more.

CIFAR-100 is the more demanding benchmark because the features required to distinguish 100 fine-grained categories are more diverse than MNIST digits. The difference between a beaver and an otter requires detecting subtle texture and shape differences that are far more complex than the stroke patterns distinguishing handwritten digits. The number of tasks is also larger (20 compared to 5 for MNIST), giving more opportunity for forgetting to accumulate over the training sequence.

Methods are typically evaluated in both task-incremental mode (task identity known at test time) and class-incremental mode (task identity unknown). The gap between these two modes reveals how much a method relies on task identity to route examples to the right classifier head. A method with a large task-IL to class-IL performance gap is essentially cheating in the easier setting: it builds a separate classifier per task and then uses the task label to select the right one at test time. This is a valid engineering solution in some applications, but it does not demonstrate continual learning of unified representations.

One important practical note about CIFAR benchmarks is that they require a feature extractor capable of handling natural images. Many papers use a pretrained ResNet backbone, which already has strong visual representations. This creates an implicit confound: the model's ability to handle multiple tasks may partly reflect the richness of the pretrained features rather than the effectiveness of the continual learning algorithm. When reporting results on CIFAR benchmarks, it is important to specify whether the backbone is pretrained and frozen, pretrained and fine-tuned, or trained from scratch, since this choice dramatically affects the absolute performance numbers and the relative advantage of different continual learning strategies.

Tiny ImageNet is a downsampled version of ImageNet with 200 classes at 64x64 resolution, selected from the original 1,000 ImageNet classes to span a diverse range of objects, animals, and scenes. Continual learning benchmarks typically split it into 10 to 40 tasks of 20 to 5 classes each. The larger number of tasks and higher visual complexity make it a considerably harder evaluation than CIFAR benchmarks.

It provides a harder visual recognition challenge than CIFAR-100 because the classes are more diverse and the intra-class variation is higher. Recognizing 200 object categories across wildly different visual domains, from fish and frogs to street furniture and sports equipment, requires representations that span a much broader feature space than the more semantically coherent CIFAR-100 categories.

Tiny ImageNet benchmarks are used to evaluate whether continual learning methods scale to more realistic inputs. Some methods that work well on MNIST and CIFAR fail to generalize to Tiny ImageNet because their memory budgets, replay strategies, or regularization penalties do not scale proportionally with task complexity. A method that stores 20 examples per class to enable replay might work well with 10 MNIST classes but require 200 stored examples per class for Tiny ImageNet to achieve the same quality of replay, and the storage cost of 200 examples times 200 classes at 64x64 resolution quickly becomes prohibitive.

Evaluating on Tiny ImageNet is therefore a useful check on scalability. A continual learning method that performs strongly on MNIST but weakly on Tiny ImageNet is likely overfitting its hyperparameters and design choices to the simpler regime. Methods that maintain their relative advantage across both scales provide stronger evidence that the advantage scales.

CORe50 is designed specifically for object recognition under domain shift, and it differs fundamentally from the class-split benchmarks in the nature of the challenge it poses. It contains 50 household objects, each belonging to one of 10 categories (plug adapters, mobile phones, scissors, light bulbs, cans, glasses, balls, markers, cups, remote controls), captured across 11 recording sessions with varying backgrounds, lighting, and viewpoints.

Unlike the class-split benchmarks where the objects and categories change from task to task, CORe50 presents distributional shift: the same objects look different in different recording sessions because of environmental variation. This is a much more realistic model of the challenges a deployed vision system would face. A robot that learns to recognize a coffee mug in a bright kitchen might be tested on recognizing the same mug in a dim hallway. The object is the same; the appearance is different.

The benchmark supports multiple evaluation protocols. In the new instances (NI) protocol, each session is a task and the object categories remain the same but the visual distribution shifts. In the new categories (NC) protocol, each task introduces new object categories and the visual distribution also changes. In the new instances and categories (NIC) protocol, both the distribution and the categories shift simultaneously, creating the most realistic and demanding evaluation scenario.

CORe50 is particularly relevant for robotics and mobile vision applications where the deployment environment changes continuously. It tests whether continual learning methods handle distributional shift, not just class-incremental splits. A method that excels at class-incremental benchmarks by building a separate classifier per class may fail entirely on CORe50's NI protocol, where the task is always recognizing the same 10 categories but under different visual conditions, because the per-class classifier approach provides no benefit when the categories do not change.

As continual learning has expanded to language models, benchmarks have emerged that sequence NLP classification tasks rather than image tasks. The most common protocols use subsets of GLUE or SuperGLUE benchmarks arranged sequentially. A model might train on sentiment analysis from one dataset, then natural language inference from another, then question answering from a third, then named entity recognition from a fourth. Each task tests different linguistic capabilities and requires different output formats and feature specializations.

Domain-specific classification sequences present a different angle on NLP continual learning. A model might learn to classify medical literature, then legal documents, then financial filings, then social media posts, all with the same text-classification output format but across wildly different vocabularies, writing styles, and domain knowledge requirements. Forgetting in this setting means losing the ability to correctly classify medical texts after adapting to financial language, even though both tasks technically use the same output labels.

Multilingual tasks presented one language at a time form yet another variant. A multilingual model might learn English, then French, then German, then Mandarin. Because languages share grammatical structures and some lexical overlap, transfer can be measured across this sequence. The question of whether early language learning transfers to later languages, and whether later language learning degrades earlier language capability, is practically important for building language models that serve diverse global users.

These NLP benchmarks reveal that large pretrained language models exhibit surprisingly strong resistance to forgetting compared to models trained from scratch. The representations learned during pretraining on massive text corpora are broad enough that fine-tuning on one task rarely completely overwrites them. Catastrophic forgetting in large language models is real but typically less severe than in smaller models trained from random initialization. This phenomenon, often called stability-plasticity in the LLM context, reflects the fact that pretrained representations resist the specific gradient updates produced by individual fine-tuning datasets. The upcoming chapters on continual learning in foundation models will examine this phenomenon in detail, including the conditions under which even large models exhibit significant forgetting and the techniques developed to address it.