Continual Learning Evaluation › Average Accuracy

The simplest scalar summary is average accuracy after the model has completed all $N$ tasks. We read the final row of the matrix (after training on all tasks) and average across all task columns:

where:

• $R_{N,j}$ is the accuracy on task $T_j$ after the model has been trained through all $N$ tasks
• $N$ is the total number of tasks

Average accuracy is easy to compute and easy to compare across methods. Its weakness is that it conflates forgetting with never learning. A method that learns every task perfectly but forgets everything old, and a method that never forgets but learns nothing new, can both produce the same average accuracy score if the initial and final performance happen to balance out. More concretely, imagine a 5-task scenario where the naive baseline achieves 95% accuracy on the final task but only 50% on the first four (chance for binary classification). The average is $0.50 \times 4/5 + 0.95 \times 1/5 = 0.59$. Now compare with a method that achieves 70% on every task. The second method's average is also $0.70$, which looks better on paper, but the underlying dynamics are completely different. One model forgets catastrophically; the other learns modestly but retains everything.

Despite this weakness, average accuracy remains the most widely reported continual learning metric because it provides a direct comparison to a natural upper bound: a joint model trained on all tasks simultaneously. The joint model sees all training data at once and is not subject to forgetting or interference. The gap between a continual learner's average accuracy and the joint model's accuracy summarizes the total cost of sequential training. A continual learning method is generally considered competitive if it closes most of this gap, even if it does not match the joint model exactly.

A related metric is average accuracy at each training step, which tracks the final average and how average performance evolves as the model absorbs new tasks. This trajectory often reveals a characteristic sawtooth pattern in naive baselines: accuracy jumps when a new task is freshly learned and then falls as subsequent training causes forgetting. Methods that successfully mitigate forgetting show a much smoother, steadily improving trajectory.