Continual Learning Evaluation › Forgetting Measure

The forgetting measure $F$ is closely related to BWT but focuses on the worst-case drop from peak to final performance, rather than comparing against the diagonal entry immediately after training. For each task, it finds the highest accuracy ever achieved on that task across all evaluation points during the training sequence, and measures how far the final accuracy has fallen from that peak.

The distinction matters because a task might continue to improve slightly after its initial training, perhaps because representations from subsequent tasks reinforce it. In that case, the diagonal entry $R_{j,j}$ would understate the task's actual best performance, and BWT would give a misleadingly optimistic picture. The forgetting measure avoids this by anchoring to the true maximum.

The per-task forgetting for task $T_j$ is:

$$F_j = \max_{k \in \{j, \ldots, N-1\}} R_{k,j} - R_{N,j}$$

where $\max_{k \in \{j, \ldots, N-1\}} R_{k,j}$ is the highest accuracy achieved on task $T_j$ at any point after the model was first trained on it, and $R_{N,j}$ is the final accuracy after all training is complete.

The aggregate forgetting measure averages over all tasks except the last (which has no subsequent training to cause forgetting):

$$F = \frac{1}{N-1} \sum_{j=1}^{N-1} F_j$$

Forgetting is always non-negative by construction, since it compares the final accuracy against the maximum ever achieved. A value of zero means the model never fell below its peak on any task. High values indicate severe degradation relative to what the model once knew.

The difference between $F$ and BWT is subtle but meaningful. BWT compares final accuracy against the accuracy right after training each task: if a task is learned poorly initially but then somehow improves through beneficial transfer from later tasks (positive BWT), BWT can still be non-negative even if later degradation occurs. The forgetting measure captures the degradation more directly by anchoring to the highest point ever reached. This makes $F$ a more conservative metric that is less susceptible to being inflated by cases where initial learning is poor but later transfer happens to help.

In practical reporting, both metrics convey useful information. BWT tells you how the model's final state compares to its state right after first learning each task, which is the most natural comparison point if the goal is to preserve knowledge from when it was freshest. The forgetting measure tells you how much of the model's all-time best performance on each task was ultimately lost, regardless of when that peak occurred. For tasks where knowledge accumulates gradually through repeated exposure, the forgetting measure is the more appropriate choice.