Regularization Methods › Elastic Weight Consolidation

Elastic Weight Consolidation (EWC), introduced by Kirkpatrick et al. (2017), derives its importance weights from the curvature of the loss landscape using the Fisher information matrix. The name comes from an analogy to elasticity: important weights are held in place by strong springs, while unimportant weights can move freely. The spring constant for each weight is proportional to its importance, so the restoring force grows linearly with displacement.

The paper that introduced EWC drew an explicit analogy to how biological neural systems achieve continual learning. Neuroscientists have long observed that synaptic connections in the brain are not equally plastic. Some synapses are stable and resist modification, while others remain highly plastic and readily respond to new experiences. This homeostatic plasticity prevents older, well-established memories from being overwritten by every new learning episode. EWC attempts to replicate this behavior computationally: high-importance parameters are the "stable synapses" and low-importance ones are the "plastic synapses."

To understand EWC's importance measure, we need to think carefully about what determines whether a parameter is important. Intuitively, a parameter is important for task $T$ if small changes to it cause large changes in the task loss. This is exactly what the second derivative of the loss surface captures. But computing the full Hessian of the loss for large neural networks is prohibitively expensive: for a network with $n$ parameters, the Hessian is an $n \times n$ matrix that requires $O(n^2)$ space and expensive computation.

The Fisher information matrix $F$ provides a principled and computationally tractable way to approximate this curvature. For a probabilistic model with parameters $\theta$, the Fisher information matrix is defined as:

$$F = \mathbb{E}_{x \sim p_{\text{data}}} \left[ \nabla_\theta \log p(x | \theta) \cdot \nabla_\theta \log p(x | \theta)^T \right]$$

where:

• $p(x | \theta)$: the model's output probability distribution over data $x$ given parameters $\theta$
• $\nabla_\theta \log p(x | \theta)$: the gradient of the log-likelihood with respect to parameters (also called the score function)
• The outer product creates a matrix capturing the covariance of the score function
• The expectation is taken over the training data distribution for the previous task

The Fisher matrix is particularly useful because it equals the expected Hessian of the negative log-likelihood under certain regularity conditions. This equality, known as the Fisher information identity, means that the curvature information you want for measuring parameter importance is exactly what the Fisher matrix encodes. The diagonal entries $F_{ii}$ therefore approximate the curvature of the loss in the direction of each parameter: a large $F_{ii}$ means the loss changes rapidly when $\theta_i$ changes, indicating high importance.

Why use the Fisher rather than the Hessian directly? The Fisher is always positive semi-definite (because it is an expectation of outer products), which means it is a valid metric on parameter space. The Hessian can be indefinite away from a local minimum, which creates problems for the regularization interpretation. More importantly, the Fisher can be estimated using only first-order information: you just need gradients, not second-order computations. This makes it computationally accessible through automatic differentiation.

EWC's derivation goes deeper than just using the Fisher as a heuristic. The method has a principled Bayesian justification that reveals why the Fisher is the right thing to use. After training on task $A$, the posterior distribution over parameters is $p(\theta | \mathcal{D}_A)$. When training on task $B$, the ideal Bayesian update would compute:

$$p(\theta | \mathcal{D}_A, \mathcal{D}_B) \propto p(\mathcal{D}_B | \theta) \cdot p(\theta | \mathcal{D}_A)$$

This equation says: the posterior over parameters given both datasets is proportional to the likelihood of the new data times the prior, where the prior is the posterior from the previous task. In other words, the old posterior plays the role of a prior for learning the new task. This is Bayesian updating, and it is the ideal way to incorporate new evidence without forgetting old evidence.

The challenge is that $p(\theta | \mathcal{D}_A)$ is generally intractable for neural networks. The posterior over a high-dimensional parameter space with a nonlinear model has no closed form, and sampling from it is computationally prohibitive. EWC approximates this posterior using a Laplace approximation: a Gaussian centered at the MAP estimate $\theta^*_A$ with covariance equal to the inverse Fisher information matrix.

$$p(\theta | \mathcal{D}_A) \approx \mathcal{N}(\theta^*_A, F_A^{-1})$$

The Laplace approximation is the simplest possible Gaussian approximation to a posterior. It treats the log-posterior as a quadratic function near its maximum (the MAP estimate), which is equivalent to fitting a Gaussian whose precision matrix (inverse covariance) equals the curvature of the log-posterior at the mode. Under the assumption that the prior is flat (or wide), the curvature of the log-posterior equals the curvature of the log-likelihood, which is precisely the Fisher information matrix.

Taking the negative log of this Gaussian prior and using it as a regularization term during training on $\mathcal{D}_B$ yields the EWC objective:

$$\mathcal{L}_{\text{EWC}}(\theta) = \mathcal{L}_B(\theta) + \frac{\lambda}{2} \sum_i F_{A,ii} \cdot (\theta_i - \theta^*_{A,i})^2$$

where:

• $\mathcal{L}_B(\theta)$: the loss on task $B$'s training data
• $F_{A,ii}$: the $i$-th diagonal entry of the Fisher information matrix computed on task $A$'s data
• $\theta^*_{A,i}$: the $i$-th parameter value at the optimum for task $A$
• $\lambda$: the EWC regularization strength

The diagonal Fisher approximation makes the method computationally tractable. Computing the full Fisher matrix requires $O(|\theta|^2)$ storage, which is infeasible for modern networks with millions of parameters. The diagonal approximation reduces this to $O(|\theta|)$ by keeping only the diagonal entries, treating each parameter independently. This independence assumption discards any correlations between parameters, which is a significant approximation, but one that makes the method scalable while retaining most of the benefit.

The Bayesian framing also clarifies what EWC is doing conceptually. Rather than adding an arbitrary regularizer, EWC performs approximate Bayesian inference. The EWC loss function is the negative log of the approximate posterior given both tasks' data. In this sense, EWC's theoretical grounding is as strong as the Laplace approximation itself, and the main source of error is the diagonal approximation to the Fisher, not the Gaussian approximation to the posterior per se.

In practice, the diagonal Fisher is approximated by averaging the squared gradients over the training data. For each parameter $\theta_i$:

$$F_{ii} \approx \frac{1}{|\mathcal{D}_A|} \sum_{(x, y) \in \mathcal{D}_A} \left( \frac{\partial \log p(y | x, \theta)}{\partial \theta_i} \right)^2$$

where:

• $\mathcal{D}_A$: the training dataset for task $A$
• $p(y | x, \theta)$: the model's predicted probability for label $y$ given input $x$
• The squared gradient approximates the curvature at the current parameter values

This computation runs after training on task $A$, requiring a forward and backward pass over the data. The result is a vector of the same dimension as $\theta$, where each entry captures the expected squared gradient for that parameter. The implementation is straightforward: run the training data through the network, compute gradients of the log-likelihood with respect to each parameter, square them, and average.

Parameters in early layers (close to the input) tend to have lower Fisher values because small perturbations there must propagate through many nonlinearities before affecting the output. By the time the perturbation reaches the final layer, the nonlinearities have dampened much of the sensitivity. Parameters in final layers often have higher values because they directly shape the prediction. This creates a natural layering of protection: the last few layers of the network are typically the most constrained by EWC, while the earlier representation layers are relatively free to adapt. Whether this pattern matches biological reality is an interesting open question, but it has important practical implications for architecture design.

When training on a third task $T_C$ after having learned $T_A$ and $T_B$, the regularization term must protect parameters important to both prior tasks. There are two common approaches, and the right choice depends on how many tasks you expect and how much memory you can afford.

Online EWC accumulates the Fisher information across tasks:

$$F_{\text{combined},ii} = F_{A,ii} + F_{B,ii}$$

This approximates the joint Fisher over all previous tasks with a single merged importance vector. The advantage is that memory requirements stay constant: you always store just one importance vector and one set of reference parameters. The disadvantage is that this approximation becomes less accurate as more tasks accumulate. Merging Fisher matrices discards the task-specific anchor points, so the combined importance vector no longer corresponds to any single set of reference parameters. In practice, researchers often maintain a single reference point at the most recent task's optimum and use the accumulated Fisher as the importance measure, accepting some approximation error.

Task-specific EWC maintains separate Fisher matrices and anchor points for each previous task:

$$\mathcal{L}_{\text{EWC}}(\theta) = \mathcal{L}_C(\theta) + \frac{\lambda}{2} \sum_{t \in \{A, B\}} \sum_i F_{t,ii} \cdot (\theta_i - \theta^*_{t,i})^2$$

This is more accurate but requires memory proportional to the number of past tasks, which becomes problematic in long task sequences. If you are learning a thousand tasks, storing a thousand Fisher vectors and anchor points is memory-intensive. Task-specific EWC is most appropriate when the number of tasks is small and known in advance.

A practical middle ground is to use online EWC with periodic consolidation: every $K$ tasks, merge the accumulated Fisher information and update the anchor point. This trades some approximation accuracy for bounded memory usage while still being more accurate than pure online accumulation.