Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Regularization Methods › Worked Example: Protecting Weights Across Two Tasks

Michael BrenndoerferFebruary 22, 20264 min read

Language AI HandbookAll chapters

Part 46 of 67Continual Learning

Chapter 353Regularization Methods › Worked Example: Protecting Weights Across Two Tasks

Reading section 6 of 10 · Worked Example: Protecting Weights Across Two Tasks · 4 minSection 6 of 10 · 4 min

Section 6 of 10 · 4 min read

Regularization Methods › Worked Example: Protecting Weights Across Two TasksOne section · Sections in this chapter6 / 10 ↓

Sections in this chapter10

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

To make the regularization intuition concrete, consider a small network trained sequentially on two classification tasks. After training on task A, we compute importance weights. When training on task B, parameters with high importance from task A are penalized for deviating from their task-A values.

Consider a single parameter $\theta_j$ with:

Task A optimum: $\theta^*_{A,j} = 0.8$
EWC importance: $F_{jj} = 5.0$ (high: this parameter matters a lot for task A)
Current value during task B training: $\theta_j = 0.5$

The EWC regularization penalty contribution from this single parameter is:

\frac{\lambda}{2} \cdot F_{jj} \cdot (\theta_j - \theta^*_{A,j})^2 = \frac{\lambda}{2} \cdot 5.0 \cdot (0.5 - 0.8)^2 = \frac{\lambda}{2} \cdot 5.0 \cdot 0.09 = 0.225\lambda

This penalty gradient pushes $\theta_j$ back toward 0.8. The gradient of the regularization term with respect to $\theta_j$ is:

\frac{\partial \mathcal{L}_{\text{reg}}}{\partial \theta_j} = \lambda \cdot F_{jj} \cdot (\theta_j - \theta^*_{A,j}) = \lambda \cdot 5.0 \cdot (0.5 - 0.8) = -1.5\lambda

This negative gradient pushes $\theta_j$ upward, toward its original value of 0.8. The total gradient for this parameter during task B training is the sum of the task B gradient and this restoring force. If task B's gradient for $\theta_j$ is small (say, $-0.1$ ), the EWC restoring force dominates and the parameter stays near its task A optimum. If task B strongly needs to change $\theta_j$ (say, its gradient is $-3.0$ ), task B's learning will partially override the protection. The interplay between the task gradient and the restoring force determines the final parameter value after each step, and the network will converge to a compromise point where neither force is fully satisfied.

Now consider a second parameter $\theta_k$ with importance $F_{kk} = 0.05$ (low: this parameter barely affected task A's loss). Even a large displacement, say $\theta_k - \theta^*_{A,k} = 2.0$ , produces only a small penalty:

\frac{\lambda}{2} \cdot 0.05 \cdot (2.0)^2 = 0.1\lambda

This parameter is essentially free to adapt to task B. The network can repurpose it entirely without incurring significant regularization cost. This is the key mechanism: the model's plasticity is redirected toward parameters that are available (low importance) while rigidity is enforced on parameters that are essential (high importance). Rather than distributing learning uniformly across all weights, EWC carves out a subspace of available parameters and confines the new task's learning to that subspace. The available subspace is exactly the set of parameters with low Fisher values for previous tasks.

The following visualization shows how EWC importance weights shape the effective learning rate across parameters. Parameters with high Fisher values face a strong restoring force that partially cancels the task B gradient; parameters with near-zero Fisher values can update freely.

Out[4]:

Visualization

Line plot showing restoring gradient magnitude vs parameter displacement for four Fisher importance levels. — EWC restoring gradient magnitude as a function of parameter displacement from the task-A anchor, for four different Fisher importance levels. High-importance parameters (Fisher = 5.0) accumulate a restoring gradient five times larger than mid-importance parameters (Fisher = 1.0) at any given displacement, strongly opposing changes that would damage task-A performance. Low-importance parameters (Fisher = 0.1) are nearly unconstrained and free to adapt to new tasks.

Parameters with Fisher value 5.0 experience a restoring gradient five times larger than those with Fisher value 1.0 for any given displacement. This multiplicative scaling is what differentiates EWC from uniform $L_2$ regularization, which would apply the same slope for all parameters. Notice also that the restoring gradient grows linearly with displacement: the further you push a protected parameter from its anchor, the harder it pushes back. This Hookean behavior is exactly the "elastic" quality that gives EWC its name.