Continual Learning Problem › A Code Walkthrough of the Forgetting Curve
Language AI HandbookAll chapters Continual Learning Problem › A Code Walkthrough of the Forgetting CurveOne section · Sections in this chapter6 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let us build a clearer picture of how forgetting unfolds by simulating the full sequential training process with accuracy tracking at each step. This walkthrough demonstrates BWT, FWT, and the progression of performance across tasks in a controlled, observable setting.
import numpy as np
from sklearn.preprocessing import StandardScaler
rng = np.random.default_rng(0)
def make_task(n_samples, center, std=0.8):
"""Generate a 2D binary classification task with Gaussian clusters."""
X_pos = rng.normal(loc=center, scale=std, size=(n_samples // 2, 2))
X_neg = rng.normal(
loc=-np.array(center), scale=std, size=(n_samples // 2, 2)
)
X = np.vstack([X_pos, X_neg])
y = np.array([1] * (n_samples // 2) + [0] * (n_samples // 2))
return X, y
# Create three tasks with increasingly different distributions
tasks = [
make_task(400, center=[2.0, 0.5]),
make_task(400, center=[-0.5, 2.0]),
make_task(400, center=[1.0, -2.0]),
]
scalers = [StandardScaler().fit(X) for X, _ in tasks]
tasks_scaled = [(scalers[i].transform(X), y) for i, (X, y) in enumerate(tasks)]# Track the accuracy matrix: acc_matrix[k][j] = accuracy on task j after training on task k
n_tasks = len(tasks)
epochs_per_task = 40
# acc_matrix[i][j] = accuracy on task j after completing training phase i
acc_matrix = np.zeros((n_tasks, n_tasks))
# Model trained right after seeing each task for the first time
immediate_acc = {}
clf = SGDClassifier(
loss="log_loss", max_iter=1, warm_start=True, random_state=0
)
for task_idx in range(n_tasks):
X_train, y_train = tasks_scaled[task_idx]
for epoch in range(epochs_per_task):
clf.partial_fit(X_train, y_train, classes=[0, 1])
immediate_acc[task_idx] = accuracy_score(y_train, clf.predict(X_train))
# Record accuracy on all tasks seen so far
for eval_task_idx in range(task_idx + 1):
X_eval, y_eval = tasks_scaled[eval_task_idx]
acc_matrix[task_idx][eval_task_idx] = accuracy_score(
y_eval, clf.predict(X_eval)
)Accuracy matrix (rows = after training phase, cols = task evaluated):
Task 1 Task 2 Task 3
After T1 0.995 --- ---
After T2 0.670 0.995 ---
After T3 0.352 0.005 1.000
Immediate accuracy on each task: ['0.995', '0.995', '1.000']
Final accuracy across all tasks: ['0.352', '0.005', '1.000']
Average final accuracy: 0.452
Backward Transfer (BWT): -0.816Reading across each row of the accuracy matrix reveals the forgetting pattern. When you look at task 1's accuracy column, it starts high (when the model just finished training on it) and progressively drops with each new task trained. By the time training on task 3 is complete, the model retains almost no performance on task 1. The backward transfer score quantifies this degradation in a single number.
The diagonal entries of the accuracy matrix, which show each task's accuracy immediately after training, are important baselines. They represent the best accuracy this model architecture can achieve on each task in isolation, given the training budget. The off-diagonal entries in the final row represent the best accuracy achievable while also maintaining knowledge of subsequent tasks. The gap between the two reveals the cost of sequential training without any continual learning mechanism.
# Compute per-epoch accuracy curves to show the exact moment of forgetting
acc_curves = {j: [] for j in range(n_tasks)}
n_epochs_total = epochs_per_task * n_tasks
clf2 = SGDClassifier(
loss="log_loss", max_iter=1, warm_start=True, random_state=0
)
task_boundaries = []
global_epoch = 0
for task_idx in range(n_tasks):
task_boundaries.append(global_epoch)
X_train, y_train = tasks_scaled[task_idx]
for epoch in range(epochs_per_task):
clf2.partial_fit(X_train, y_train, classes=[0, 1])
for j in range(n_tasks):
X_eval, y_eval = tasks_scaled[j]
acc_curves[j].append(accuracy_score(y_eval, clf2.predict(X_eval)))
global_epoch += 1
The plot in @fig-catastrophic-forgetting-curves makes forgetting viscerally clear. At each dashed task boundary, at least one earlier task loses accuracy abruptly. Task 1 retains some of what it learned after Task 2 but degrades further after Task 3, while Task 2 falls from near-perfect accuracy to almost zero as soon as Task 3 training begins. The damage occurs at the distribution shift rather than as a slow decay, which is the "catastrophic" aspect of catastrophic forgetting.
Notice also what happens to the future tasks before training on them begins. Their accuracy hovers around chance level during earlier task phases, because the model has not been exposed to their data at all. Once training on each task begins, accuracy rises rapidly, which represents normal learning rather than anything unusual. The problem is entirely in what happens to earlier tasks when this learning takes place.