Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Continual Learning Problem › Worked Example: Measuring Forgetting on a Two-Task Problem

Michael BrenndoerferFebruary 19, 20264 min read

Language AI HandbookAll chapters

Part 46 of 67Continual Learning

Chapter 352Continual Learning Problem › Worked Example: Measuring Forgetting on a Two-Task Problem

Reading section 8 of 11 · Worked Example: Measuring Forgetting on a Two-Task Problem · 4 minSection 8 of 11 · 4 min

Section 8 of 11 · 4 min read

Continual Learning Problem › Worked Example: Measuring Forgetting on a Two-Task ProblemOne section · Sections in this chapter8 / 11 ↓

Sections in this chapter11

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let us work through a concrete numerical example to solidify the metrics and their interpretation. Suppose we have two tasks and a model whose accuracy evolves as follows:

After training on task 1: accuracy on task 1 is 0.88. After training on task 2: accuracy on task 1 drops to 0.52, accuracy on task 2 is 0.84.

We can compute:

\text{BWT} = a_{2,1} - a_{1,1} = 0.52 - 0.88 = -0.36

where:

$a_{2,1}$ : accuracy on task 1 after training on task 2 (final evaluation)
$a_{1,1}$ : accuracy on task 1 immediately after training on it

A BWT of $-0.36$ means that completing training on task 2 caused a 36-percentage-point drop in task 1 performance. This is severe forgetting. Any BWT worse than $-0.10$ in a real deployment would be a serious quality problem for most applications.

For forward transfer, suppose before training on task 2 at all, the model (from its task 1 training) already achieves 0.62 on task 2, and a random baseline achieves 0.50:

\text{FWT} = a_{1,2} - b_2 = 0.62 - 0.50 = 0.12

where:

$a_{1,2}$ : accuracy on task 2 evaluated with the task-1-trained model, before any task 2 training
$b_2$ : random baseline accuracy on task 2

A positive FWT of 0.12 means that learning task 1 provided a 12-percentage-point head start on task 2, suggesting positive transfer of shared representations. This is an encouraging sign: the two tasks share enough structure that learning one helps with the other. In language model settings, positive forward transfer is common because many NLP tasks share underlying linguistic structure. A model that learned to recognize sentiment may generalize somewhat to detecting subjectivity even before seeing subjectivity labels.

The average final accuracy across both tasks is:

\text{Avg} = \frac{0.52 + 0.84}{2} = 0.68

This average is pulled down substantially by the forgetting on task 1. A good continual learning method would aim to keep average accuracy closer to the individual task peaks (around 0.86 in this example) by preventing BWT from becoming so negative. The ideal outcome would be something like: task 1 accuracy stays at 0.87 after task 2 training, and task 2 accuracy reaches 0.84 as before. Average accuracy would then be 0.86, and BWT would be approximately $-0.01$ : essentially zero forgetting with full learning.

In[16]:

Code

# Numerical verification of BWT, FWT, and Avg formulas
a_1_1 = 0.88  # accuracy on task 1 after training on task 1
a_2_1 = 0.52  # accuracy on task 1 after training on task 2
a_2_2 = 0.84  # accuracy on task 2 after training on task 2
a_1_2 = (
    0.62  # accuracy on task 2 before any training on it (using task-1 model)
)
b_2 = 0.50  # random baseline accuracy on task 2

bwt = a_2_1 - a_1_1
fwt = a_1_2 - b_2
avg_acc = (a_2_1 + a_2_2) / 2

Out[17]:

Console

Backward Transfer (BWT):  -0.36  (negative = forgetting)
Forward Transfer (FWT):   +0.12  (positive = positive transfer)
Average final accuracy:   0.68

The output confirms our manual calculation. The combination of negative BWT and positive FWT tells a coherent story: the model is learning tasks in a way that enables some positive transfer (the tasks share useful features), but sequential gradient updates are still destructive enough to cause severe forgetting. A continual learning method that could preserve the positive FWT while bringing BWT close to zero would be a substantial improvement.

This pattern, strong forward transfer combined with severe backward interference, is common in practice and suggests that the solution should focus on protecting previously learned parameters rather than on how new tasks are initially learned. The new task learning itself is working well (FWT is positive); what needs improvement is the mechanism that prevents this learning from overwriting old knowledge.