Catastrophic Forgetting › Causes at the Parameter Level

To understand forgetting deeply, we need to examine what happens to individual parameters during fine-tuning. The aggregate measures of forgetting we discussed earlier emerge from microscopic changes happening simultaneously across millions of weights. By understanding these parameter-level dynamics, we can design more targeted interventions that protect the right parameters without unnecessarily constraining the ones that can safely change.

The key insight at the parameter level is that not all weights are equal. Some encode general linguistic knowledge that underlies many capabilities. Others encode task-specific patterns that are relevant only in narrow contexts. Still others are relatively uncommitted, serving as general-purpose capacity that the model uses flexibly depending on what it is asked to do. Catastrophic forgetting happens primarily when fine-tuning updates the first category, the general linguistic knowledge weights, in service of the second category, the task-specific patterns.

Not all parameters contribute equally to each task. Some weights are necessary for general language understanding, carrying information about syntax and semantics that applies broadly. Others specialize in specific capabilities, encoding task-specific patterns that are only relevant in certain contexts. Many are relatively unimportant for any particular function, serving as "slack" capacity that the model uses flexibly.

The challenge is that fine-tuning doesn't distinguish between these categories; it updates all parameters to reduce the fine-tuning loss. The gradient provides information about how changing each parameter would affect the fine-tuning objective, but it provides no information about how those same changes would affect other objectives. A parameter necessary for named entity recognition receives the same treatment as a parameter that is entirely unimportant for prior capabilities.

Think of parameter importance as a map showing which areas of a city must not be excavated. Some zones contain necessary infrastructure: main water mains, high-voltage power lines, fiber trunks. Other zones are safe to dig in without consequences. Standard fine-tuning applies the same excavation force everywhere, with no knowledge of which zones are necessary. The mitigation strategies we will discuss are essentially different ways to generate this importance map and use it to protect the necessary zones.

The key insight is that parameters in the "original task necessary" category (blue points) need protection during fine-tuning. If these weights change significantly, original capabilities degrade, even if the changes help the fine-tuning objective. The scatter plot reveals the geometry of this problem: parameters occupy different regions depending on their importance to each task, and the region that matters most for forgetting is the one containing parameters that are important for original capabilities but not for the fine-tuning task.

When gradients for the fine-tuning task oppose what would maintain original capabilities, we have gradient interference. This phenomenon occurs when the direction that improves the fine-tuning objective is exactly opposite to the direction that would preserve prior performance.

Consider a weight that needs to increase to improve sentiment classification but whose original value supported named entity recognition. Standard gradient descent simply follows the fine-tuning gradient, potentially destroying NER performance. The optimizer has no way to know that this particular weight is important for a capability we want to preserve. It only sees the gradient pointing upward and dutifully moves the weight in that direction.

Gradient interference is particularly insidious because it is invisible during training. The training loss decreases, validation accuracy on the fine-tuning task improves, and everything appears to be working correctly. Only when we evaluate on prior tasks do we discover that the gradients we followed led us away from the region of weight space that supported those capabilities. This delayed discovery is one reason why catastrophic forgetting often goes undetected until production deployment reveals failures in unexpected contexts.

We can characterize the relationship between the fine-tuning gradient $\mathbf{g}_{\text{new}}$ and the gradient that would maintain prior capability $\mathbf{g}_{\text{prior}}$ through their dot product. When the dot product is positive, the gradients are aligned and learning the new task is compatible with preserving the old one. When it is zero, the gradients are orthogonal and updating for the new task neither helps nor hurts the old one. When it is negative, the gradients conflict directly and updating for the new task actively degrades the old capability:

$$\text{interference} = -\frac{\mathbf{g}_{\text{new}} \cdot \mathbf{g}_{\text{prior}}}{\|\mathbf{g}_{\text{new}}\| \|\mathbf{g}_{\text{prior}}\|}$$

where:

• $\mathbf{g}_{\text{new}}$: the gradient vector computed from the fine-tuning task loss, pointing in the direction that improves new task performance
• $\mathbf{g}_{\text{prior}}$: the gradient vector that would improve or maintain performance on prior tasks, computed if we had access to prior task data
• $\mathbf{g}_{\text{new}} \cdot \mathbf{g}_{\text{prior}}$: the dot product between the two gradient vectors, measuring their directional alignment
• $\|\mathbf{g}_{\text{new}}\| \|\mathbf{g}_{\text{prior}}\|$: the product of the gradient magnitudes, normalizing the dot product to the range $[-1, 1]$
• The negative sign: ensures that positive interference values indicate harmful conflict, matching the intuition that high interference leads to high forgetting

Why does this formula make sense? The normalized dot product (cosine similarity) ranges from $-1$ (perfectly aligned, beneficial) to $+1$ (perfectly opposed, maximally harmful). By negating it, we get a measure where positive values signal danger: the new task gradient actively conflicts with preserving prior capabilities. In practice, we cannot compute $\mathbf{g}_{\text{prior}}$ without access to prior task data, which is why proxy measures like parameter importance are used instead. But the geometry illuminates why simply training on a new task can so efficiently destroy prior capabilities: when the relevant gradients point in opposite directions, no amount of careful learning rate tuning can prevent the conflict.