Continual Learning Evaluation › Continual Learning Evaluation

When a model learns a new task, how do you know whether it helped or hurt? That question turns out to be surprisingly difficult to answer. Standard machine learning evaluation compares a model's accuracy on a held-out test set after training is complete. Continual learning evaluation must do something harder: track how well a model retains what it knew before, measure what it gains from new experience, and assess whether the two goals work against each other.

The difficulty is not merely computational. It is conceptual. Evaluating a continual learner requires you to reason simultaneously about three axes that standard evaluation collapses into one. First, there is the question of retention: does the model still know what it used to know? Second, there is the question of acquisition: does the model learn new tasks as well as it would if it were trained on them in isolation? Third, there is the question of transfer: does experience with earlier tasks make later learning easier, or does it get in the way? A single accuracy number on the most recent task reveals nothing about the first two axes and conflates the second and third.

This chapter covers the metrics and protocols that make continual learning measurable. You will see why overall accuracy alone is insufficient, how forward transfer and backward transfer capture the memory-plasticity tradeoff at the heart of continual learning, and which benchmark suites the research community uses to compare methods. Together, these evaluation tools let you distinguish a model that adapts from one that simply forgets. By the end, you will have a complete quantitative vocabulary for characterizing any continual learning system, and a clear sense of where each benchmark and metric is most informative.