Continual Learning Evaluation › Evaluation Protocols

Beyond the specific benchmarks, the research community distinguishes several evaluation protocols that affect what continual learning methods can and cannot do during inference. The choice of protocol is as consequential as the choice of benchmark, and comparing methods evaluated under different protocols is typically invalid.

In task-incremental learning (task-IL), the model knows the task identity $t$ at both training and test time. This allows the model to use a separate output head for each task, routing test examples to the right classifier automatically. Because the model never needs to distinguish between tasks, the problem reduces to ensuring that learning task $t+1$ does not damage the features needed by the classifier for task $t$.

Task-IL is the most favorable setting for continual learning methods. It simplifies the problem considerably because the output ambiguity that makes class-IL difficult is removed entirely. The model knows which task is being evaluated and can apply the corresponding output head directly. Many methods that fail in class-IL settings perform well in task-IL because the only challenge remaining is weight interference in shared feature layers, not output disambiguation.

Task-IL is unrealistically easy in many practical scenarios because knowing the task identity at test time is not always available. In a real deployment, the system may not receive explicit signals about which task a given input belongs to. A question-answering system deployed in production does not know whether an incoming query belongs to the "factual lookup" task it was trained on first or the "reasoning" task it was trained on second. For this reason, task-IL results serve as an important upper bound on what can be achieved with a given architecture and training procedure, but they should not be reported as evidence of practical continual learning performance without clearly noting that task identity is provided at test time.

In class-incremental learning (class-IL), the model knows the task identity during training but not at test time. The model must output the correct class from the union of all classes seen so far, without any hint about which task a test example belongs to.

This is significantly harder than task-IL. Without the task identifier, the model must solve an implicit task discrimination problem on top of the classification problem. If the model has learned to classify digits 0 through 1 in task 1 and digits 2 through 3 in task 2, a class-IL test requires the model to correctly identify the digit from a pool of all four classes (0, 1, 2, 3) without knowing whether the image belongs to task 1 or task 2. Naive fine-tuning methods fail badly here because the latest task's output head dominates all predictions. Methods must either store explicit examples for replay, build task-agnostic representations, or employ architectural tricks to perform well.

Class-IL is the standard setting for most published continual learning comparisons because it represents a realistic scenario: a deployed model receives new data over time and must continue classifying correctly for all classes it has ever seen. A spam filter that learns to recognize a new type of phishing email should continue working correctly on all previously known spam categories as well as on the new category. The inability to provide a "this email might be of type X" hint at test time means the model must handle the entire accumulated output space simultaneously.

One important nuance in class-IL evaluation is how the output head is structured. Some methods grow the output layer as new classes arrive (expanding head), while others fix the output size from the beginning (fixed head). Expanding heads require knowing the total number of eventual classes in advance or performing dynamic architecture modification during training. Fixed heads require the model to partition a fixed output space across tasks. These architectural choices interact significantly with the continual learning method and must be specified when reporting class-IL results.

In domain-incremental learning (domain-IL), the output space is fixed but the input distribution shifts. For example, a sentiment classifier trained on movie reviews must then learn to classify restaurant reviews, then tech reviews, using the same positive/negative output labels throughout. The model does not know which domain a test example comes from.

Domain-IL is relevant for deployed systems that encounter distribution shift over time without explicitly adding new output categories. It tests whether the model builds domain-invariant features or whether its features become too specialized to the training distribution. A model that is highly specialized to the vocabulary and writing style of movie reviews may perform poorly on restaurant reviews even though the underlying task, distinguishing positive from negative sentiment, is the same.

The CORe50 NI protocol is a visual example of domain-IL: the same object categories exist throughout, but the recording sessions (domains) change. The challenge is entirely distributional, not categorical. Domain-IL is generally easier than class-IL for methods that work by expanding output capacity, since no new output heads are needed. However, it is harder for methods that rely on detecting shifts in the output distribution to trigger replay or regularization, since the output distribution is identical across domains by construction.

One underappreciated aspect of evaluation protocol is how often the accuracy matrix is measured. Some papers evaluate only at the end of training on each task (coarse-grained). Others evaluate after every gradient update (fine-grained). The choice affects what you can infer about forgetting dynamics and makes a substantial difference in the cost of running the evaluation.

Fine-grained evaluation reveals whether forgetting is gradual or catastrophic, whether it happens mostly at the beginning of a new task's training or continues throughout, and whether certain training checkpoints are particularly vulnerable. A model might lose most of its performance on task 1 during the first hundred gradient steps of task 2 training, and then maintain a stable (if degraded) level for the remaining training steps. This would appear as a single large drop in coarse-grained evaluation. Fine-grained evaluation would reveal the dynamics of that drop and potentially identify an early stopping point that balances new-task learning against old-task retention.

Coarse-grained evaluation is sufficient for comparing final performance but misses these dynamics entirely. For benchmarking purposes, end-of-task evaluation is standard because it reduces the computational cost and simplifies reporting. However, for debugging a new method, fine-grained tracking of the accuracy matrix is far more informative. Observing where in training each task's performance collapses, and how quickly, can reveal whether the problem is a learning rate that is too high, a regularization coefficient that is insufficiently calibrated, or a replay strategy that does not sample old examples frequently enough.

The evaluation frequency also affects the values of BWT and the forgetting measure. With coarse-grained evaluation, you only have access to the diagonal entries of the accuracy matrix and the final row. With fine-grained evaluation, you have access to every row, which gives you the full trajectory of each task's accuracy and allows the forgetting measure to anchor at the true peak rather than just at the end of each task's initial training phase.