Continual Learning Evaluation › The Accuracy Matrix

The foundation of continual learning evaluation is a matrix of test accuracies recorded throughout training. After training on each task $T_i$, you evaluate the model on every task $T_j$ seen so far. This produces an accuracy matrix $R$, where entry $R_{i,j}$ is the accuracy on task $T_j$ immediately after training on task $T_i$.

To understand what this matrix captures, consider a five-task example. Immediately after finishing task $T_1$, you record a single row: the model's accuracy on $T_1$ itself. After finishing $T_2$, you record two values: accuracy on $T_2$ (the new task) and accuracy on $T_1$ (which may have degraded). The matrix grows one row at a time:

where:

• $R_{i,j}$ is the test accuracy on task $T_j$ after the model has been trained sequentially through task $T_i$
• $R_{i,i}$ (the diagonal) captures immediate performance right after the model finishes learning task $T_i$
• $R_{i,j}$ for $j < i$ (lower-left entries) captures how well the model retains task $T_j$ after subsequent training on tasks $T_{j+1}, \ldots, T_i$
• $N$ is the total number of tasks

The diagonal entries represent the model at its best for each task in the sequential setting. The off-diagonal entries track degradation. A matrix with high diagonal values and off-diagonal values close to those diagonal values indicates excellent retention. A matrix where off-diagonal values collapse toward chance indicates catastrophic forgetting.

Reading the accuracy matrix provides an immediate visual diagnosis. If the matrix looks nearly uniform across every row for a given column, the model has maintained consistent performance on that task throughout training. If the entries in a column drop sharply from left to right (that is, from earlier rows to later ones), the model is forgetting that task as it learns new ones. If the diagonal entries are all high but the off-diagonal entries drop, that is the signature of a model that learns each task perfectly in isolation but loses what it once knew. If the diagonal entries themselves are lower than expected, the model has difficulty learning new tasks, perhaps because prior weight adjustments have made certain regions of parameter space harder to reach.

The matrix is the raw material from which every summary metric is derived. Rather than discarding all this information into a single number, the matrix preserves the full history of the model's performance trajectory. When you want to understand why a method works or fails, you return to the matrix and examine which entries change and by how much.