Continual Learning Evaluation › Summary

Continual learning evaluation measures how well a model retains old knowledge while acquiring new knowledge. The key concepts in this chapter are:

• The accuracy matrix $R$ is the foundation: entry $R_{i,j}$ records performance on task $T_j$ after training through task $T_i$. All summary metrics derive from this matrix, and examining it directly reveals the qualitative character of a method's behavior.
• Average accuracy after all tasks summarizes overall performance but conflates retention and learning. It should always be supplemented with more diagnostic metrics.
• Backward transfer (BWT) measures how much subsequent training degrades earlier task performance. Negative values indicate forgetting; values near zero indicate strong retention. BWT is the most direct measure of catastrophic forgetting.
• Forward transfer (FWT) measures how much earlier tasks help or hinder learning new tasks. Positive values indicate beneficial transfer; negative values indicate task interference. FWT captures whether the model is becoming a better learner over time.
• Forgetting measure captures the worst-case drop from peak to final performance on each task. It is more conservative than BWT and anchors to the true maximum rather than the post-training diagonal.
• Intransigence captures how much prior training makes it harder to absorb new tasks. It is the flip side of forgetting and is particularly important for evaluating regularization-based methods that protect old knowledge at the cost of new learning.
• Standard benchmarks (permuted MNIST, split CIFAR, CORe50, NLP sequences) provide controlled settings for comparing methods. Each benchmark tests different aspects of forgetting and transfer, and evaluating on multiple benchmarks provides stronger evidence than any single one.
• Evaluation protocols (task-IL, class-IL, domain-IL) define what task information is available at test time. Class-IL is the standard and hardest setting. Task-IL results should be clearly labeled as using privileged task identity information.
• Every metric and benchmark has limitations. Honest evaluation reports the full metric suite, specifies the exact evaluation protocol, and acknowledges what assumptions about data access and task boundaries are embedded in the comparison.

With these tools, you can characterize any continual learning method beyond a single accuracy number and understand exactly where it succeeds and fails in the memory-plasticity tradeoff. The metrics are the vocabulary for a precise technical conversation about a continual learning system's behavior, and the benchmarks are the shared language that makes comparisons across the research community meaningful.