Continual Learning Evaluation › Backward Transfer

Backward transfer (BWT) measures how much learning new tasks affects performance on previously learned tasks. The question it answers is: after the entire learning sequence is complete, how does each task's final accuracy compare to the accuracy it had right after being trained?

Formally, we compare the final row of the accuracy matrix against the diagonal:

where:

• $R_{N,j}$ is the accuracy on task $T_j$ after the model has finished training on all $N$ tasks
• $R_{j,j}$ is the accuracy on task $T_j$ right after the model finished training on it (the diagonal entry, representing peak accuracy for that task in the sequential setting)
• $N$ is the total number of tasks

The numerator for each task, $R_{N,j} - R_{j,j}$, is the change in accuracy from when the model was freshly trained on that task to when all subsequent training is complete. Negative values mean the model lost accuracy over time; positive values mean the model somehow improved on a task after no longer training on it.

Backward transfer is typically negative. A value of $-0.20$ means that, on average, tasks lose 20 percentage points of accuracy after the model learns subsequent tasks. A value near zero indicates strong retention. A positive value would mean that subsequent tasks improved performance on earlier ones, which occasionally happens through beneficial weight sharing when tasks are closely related, such as when two tasks both require detecting low-level visual features like edges or textures.

The key insight backward transfer provides is that it isolates the retention problem from the learning problem. You can have excellent average accuracy with very poor BWT if the model happens to learn later tasks well enough to compensate for forgetting early ones. Separating these effects reveals the true character of a continual learning method. A method that achieves high average accuracy through strong BWT is doing something fundamentally different from one that achieves the same average accuracy through high initial performance on the current task at the expense of everything prior.

In practice, BWT values tend to cluster in a few interpretable ranges. A BWT close to zero (within a few percentage points) indicates that the method handles forgetting well. Approaches like parameter regularization (EWC, SI) and replay-based methods typically achieve BWT values between $-0.05$ and $-0.15$ on standard benchmarks, while still accommodating new tasks.

A strongly negative BWT (below $-0.30$) signals catastrophic forgetting. A naive fine-tuning baseline on permuted MNIST often achieves BWT around $-0.70$ or worse: the model's performance on early permutations collapses almost entirely as it adapts to later permutations. On Split CIFAR-100, where tasks involve distinguishing more semantically distinct categories, naive fine-tuning BWT can be even more extreme because there is less incidental weight overlap between tasks.

Positive BWT is rare but real. It occurs when tasks share structure that reinforces each other, for instance when both tasks require detecting edges or recognizing phonemes. When you observe positive BWT, it suggests that the tasks in the benchmark are more related than typical continual learning scenarios assume, which may limit the benchmark's ability to stress-test forgetting. Researchers who observe positive BWT should look carefully at whether their benchmark requires the model to update weights in conflicting directions across tasks, or whether the tasks are so similar that learning one simply refines representations useful for the other.

One subtlety in interpreting BWT is the choice of reference point. Standard BWT compares final accuracy against the diagonal of the accuracy matrix, which is the accuracy right after training on each task in the sequential setting. This is different from comparing against the isolated baseline $b_j$. A method could achieve BWT near zero by learning each task poorly initially (low diagonal) but maintaining that low performance consistently. This would look like good retention even though the model never learned the tasks well. For this reason, BWT should always be reported alongside the diagonal values themselves or alongside average accuracy, not in isolation.