Regularization Methods › Synaptic Intelligence

Synaptic Intelligence (SI), introduced by Zenke et al. (2017), takes a fundamentally different approach to measuring parameter importance. Rather than computing importance from the loss curvature after each task is complete, SI accumulates importance online during training itself. The gradient computation you are already doing in every training step is mined for additional information about which parameters are contributing to learning.

The biological motivation for SI is distinct from EWC's. While EWC is inspired by homeostatic plasticity (stable versus plastic synapses), SI draws from the observation that biological synapses appear to track their own history of activity and use this history to modulate plasticity. Synapses that have recently been active in strengthening memory traces become more resistant to subsequent modification. SI attempts to implement this memory trace computationally.

EWC measures importance by asking: how much does the loss change if we perturb parameter $\theta_i$ at its final value? SI instead asks: how much total contribution did parameter $\theta_i$ make to reducing the loss during the entire training trajectory?

This is a fundamentally different question. EWC cares about the geometry of the loss field at the final point. SI cares about the dynamics of the optimization process. A parameter could be in a flat region of the loss at convergence (low EWC importance) but have played a central role in the optimization path that led there (high SI importance). Conversely, a parameter might be in a high-curvature region at convergence but barely moved during training (high EWC, low SI). Both are valid notions of "importance," but they capture different aspects of the parameter's role in the learned task.

Formally, SI estimates importance as the integral of the product of the gradient and the parameter change along the optimization path. During training on task $T_k$, for each parameter $\theta_i$, SI tracks:

$$\omega_i^{(k)} = -\int_{t_{\text{start}}}^{t_{\text{end}}} \frac{\partial \mathcal{L}_{T_k}}{\partial \theta_i} \cdot \dot{\theta}_i \, dt$$

where:

• $\dot{\theta}_i = d\theta_i/dt$: the rate of change of parameter $\theta_i$ during optimization
• $\frac{\partial \mathcal{L}_{T_k}}{\partial \theta_i}$: the gradient of the task loss with respect to $\theta_i$
• The negative sign makes the integral positive when gradients and updates point in the same direction (i.e., when the update reduces the loss)
• The integral is over the entire optimization trajectory from initialization to convergence for task $T_k$

A parameter with high $\omega_i^{(k)}$ was responsible for large loss reductions during training on task $T_k$, which makes it important to preserve. The intuition is that this parameter was "doing work" during optimization: it was moving in a direction aligned with the gradient, and that movement was consequential for reducing the loss.

In practice, since we optimize with discrete gradient steps rather than a continuous trajectory, $\omega_i^{(k)}$ is approximated by summing over training steps:

$$\omega_i^{(k)} \approx -\sum_{s=1}^{S} g_i^{(s)} \cdot \Delta\theta_i^{(s)}$$

where:

• $g_i^{(s)}$: the gradient of parameter $\theta_i$ at training step $s$
• $\Delta\theta_i^{(s)} = \theta_i^{(s+1)} - \theta_i^{(s)}$: the change in parameter $\theta_i$ at step $s$
• $S$: the total number of training steps for task $T_k$

The name "path integral" comes from the fact that you are integrating a quantity along the optimization path through parameter space. This is analogous to computing the work done by a force along a trajectory in physics. Just as a force that consistently pushes in the direction of movement does more work than one that pushes sideways, a gradient that consistently points in the direction of parameter updates contributes more to the path integral and thus yields higher importance.

Notice that the path integral depends on the specific trajectory taken by the optimizer, not just on the start and end points. This affects how SI estimates should be interpreted. Two different optimizers (say, SGD and Adam) that converge to the same final parameters will generally produce different SI importance estimates because they take different paths through parameter space. Adam's adaptive learning rates alter the effective direction of each step, changing which gradients align most strongly with which parameter updates. This optimizer-dependence is both a strength and a weakness: it makes SI more sensitive to the actual optimization dynamics, but it also makes the importance estimates harder to interpret theoretically.

After training on task $T_k$, SI computes the final importance measure for parameter $\theta_i$ by normalizing the accumulated gradient-update product:

$$\Omega_i^{(k)} = \frac{\omega_i^{(k)}}{(\Delta_i^{(k)})^2 + \epsilon}$$

where:

• $\omega_i^{(k)}$: the accumulated gradient-update product for task $T_k$
• $\Delta_i^{(k)} = \theta_i^{*,k} - \theta_i^{*,k-1}$: the total parameter change from the start to end of task $T_k$
• $\epsilon$: a small constant (e.g., 0.1) added for numerical stability to prevent division by zero
• The denominator normalizes by the squared total displacement, making the measure scale-invariant

This normalization is important for the right reasons. A parameter that changed a lot but also had large gradients throughout should not necessarily be considered more important than one that changed little but always pointed toward the minimum. Dividing by the squared total displacement removes the scale factor of movement, giving you something closer to "per unit of displacement, how much gradient work was done?" The ratio captures the efficiency of each parameter's contribution to learning instead of only its raw magnitude.

The $\epsilon$ in the denominator serves a secondary purpose beyond numerical stability. It provides a floor on the denominator, which prevents parameters that moved very little (near-zero $\Delta_i^{(k)}$) from receiving artificially large importance values. A parameter that was frozen by the optimizer should not be marked as maximally important just because it has a small denominator. The value of $\epsilon$ acts as a threshold: parameters whose total movement is much smaller than $\sqrt{\epsilon}$ get little credit even if their gradient-update products were large.

For continual learning across multiple tasks, SI accumulates importance by summing across all previous tasks:

$$\Omega_i = \sum_{k=1}^{K-1} \Omega_i^{(k)}$$

The total importance grows with each new task that relied on parameter $\theta_i$. This has a natural interpretation: a parameter that has been important for many past tasks should be heavily protected when learning future ones. Unlike task-specific EWC, this accumulated importance does not require storing separate importance vectors per task. You maintain a single vector that grows more refined as more tasks are processed, because importance from earlier tasks is already baked into the accumulation.

With importance weights computed, the SI regularization loss takes the same weighted quadratic form as EWC:

$$\mathcal{L}_{\text{SI}}(\theta) = \mathcal{L}_{T_k}(\theta) + c \cdot \sum_i \Omega_i \cdot (\theta_i - \theta^*_i)^2$$

where:

• $c$: the regularization strength hyperparameter (analogous to $\lambda$ in EWC)
• $\theta^*_i$: the parameter value at the start of the current task (i.e., the endpoint of the previous task)
• $\Omega_i$: the accumulated importance from all previous tasks

The practical advantage of SI is that $\omega_i^{(k)}$ is computed incrementally during training, requiring just two additional scalar quantities per parameter: the running sum and the current parameter value at task start. No separate backward pass over task data is needed after training, unlike EWC which requires re-running the training data to compute the Fisher. For scenarios where the training data cannot be stored or accessed again after training (which is common in privacy-sensitive applications), SI's online accumulation is a significant practical advantage.

The downside is implementation complexity. You must carefully instrument the training loop to accumulate gradient-update products at every step, and you must correctly reset the accumulators at task boundaries. Any bug in this accounting will silently produce incorrect importance estimates, leading to more or less forgetting than expected. EWC's importance computation is a clean, separate pass that is easier to validate in isolation.