AdaLoRA › Importance Scoring

The heart of AdaLoRA is deciding which singular value triplets to keep and which to prune. This requires an importance score that shows each component's real contribution to model performance. Designing a good importance score turns out to be more subtle than it might first appear.

Think of importance scoring as trying to identify which employees on a team are truly needed. You have two types of information available: how much work each employee is currently creating (magnitude), and how much the team's output would suffer if they were suddenly absent (sensitivity). An employee who produces a lot but would be easily replaced by others contributes less than their output suggests. An employee who produces less but is an irreplaceable coordinator, whose absence would cascade into major disruptions, is more important than their direct output implies. The best measure of importance combines both signals.

In the context of singular value triplets, the magnitude corresponds to $|\lambda_i|$: larger singular values contribute more to the weight update's norm. The sensitivity corresponds to how much the training loss would increase if that triplet were zeroed out. Together, these signals identify components that are both currently contributing and would be missed if removed.

A naive approach would rank components by $|\lambda_i|$ alone: larger singular values contribute more to the weight update's magnitude. This makes intuitive sense because the singular value directly scales the rank-1 contribution. However, magnitude alone misses important information about gradient flow. A large singular value with near-zero gradients is stable and well-fitted: it has found its optimal value and further gradient-driven changes would not improve performance. A smaller singular value with large gradients is actively being adjusted and may be necessary for continued learning, representing a component that the optimization process is still working to refine.

Consider two extremes. A singular value of 1.0 with gradient magnitude 0.001 has high magnitude but low sensitivity: removing it would change the weight update substantially, but the gradient signal suggests the loss is not currently sensitive to this component's value. A singular value of 0.1 with gradient magnitude 0.5 has low magnitude but high sensitivity: removing it would cause a small direct change to the weight update, but the large gradient indicates the loss is highly responsive to this component and pruning it would cause significant performance degradation.

AdaLoRA combines both signals into an importance score that captures current contribution and learning dynamics simultaneously:

$$S_i = \bar{s}_i \cdot |\lambda_i|$$

where:

• $S_i$: the importance score for the $i$-th singular value triplet
• $\bar{s}_i$: the smoothed sensitivity estimate derived from gradient information
• $|\lambda_i|$: the absolute magnitude of the singular value

The product structure ensures that both factors must be substantial for a component to receive a high importance score. A component is considered important only if it both currently contributes to the weight update (non-trivial magnitude) and the model would suffer if it were removed (non-trivial sensitivity). Components that are large but insensitive, or sensitive but small, receive lower scores than components that are both large and sensitive. This combination balances two complementary views of importance and guards against the failure modes of either metric alone.

The dashed contour lines in the plot stand for iso-importance curves: all components along a given contour have the same importance score. A component at (magnitude=1.0, sensitivity=0.3) has the same importance score as a component at (magnitude=0.3, sensitivity=1.0). Components below the lowest contour line are prime pruning candidates regardless of which dimension they fail on. The plot also shows that the "Prune" region in the lower-left corner, where both magnitude and sensitivity are small, clearly deserves pruning, while the upper-right region contains the components worth retaining.

The sensitivity $s_i$ for singular value $\lambda_i$ measures how much the training loss would change if that component were removed. We compute it using a first-order Taylor approximation of the loss function.

To formalize this, consider the loss $\mathcal{L}$ as a function of the singular value $\lambda_i$. We want to estimate the loss change $\Delta \mathcal{L}$ that would result from setting $\lambda_i$ to zero (pruning the component). The first-order Taylor expansion around the current value $\lambda_i^{(t)}$ gives:

$$\mathcal{L}(\lambda_i = 0) \approx \mathcal{L}(\lambda_i^{(t)}) + \frac{\partial \mathcal{L}}{\partial \lambda_i^{(t)}} \cdot (0 - \lambda_i^{(t)})$$

The change in loss from pruning is therefore:

$$\begin{aligned} \Delta \mathcal{L} &= \mathcal{L}(\lambda_i = 0) - \mathcal{L}(\lambda_i^{(t)}) \\ &\approx \frac{\partial \mathcal{L}}{\partial \lambda_i^{(t)}} \cdot (0 - \lambda_i^{(t)}) \\ &= -\lambda_i^{(t)} \cdot \frac{\partial \mathcal{L}}{\partial \lambda_i^{(t)}} \end{aligned}$$

Taking the absolute value (since we care about the magnitude of the change, not its direction) gives the instantaneous sensitivity estimate:

$$s_i^{(t)} = \left| \lambda_i^{(t)} \cdot \frac{\partial \mathcal{L}}{\partial \lambda_i^{(t)}} \right|$$

where:

• $s_i^{(t)}$: the instantaneous sensitivity at training step $t$, estimating how much the loss would increase if component $i$ were pruned
• $\lambda_i^{(t)}$: the value of the singular value at step $t$
• $\frac{\partial \mathcal{L}}{\partial \lambda_i^{(t)}}$: the gradient of the training loss with respect to $\lambda_i$ at step $t$

This quantity is sometimes called "saliency" in the neural network pruning literature, inspired by analogous formulas used for identifying important neurons and connections. We do not need to evaluate what would happen if we removed the component: we can estimate the effect using only quantities that are already computed during the standard backward pass. The gradient $\frac{\partial \mathcal{L}}{\partial \lambda_i}$ is computed anyway as part of updating $\Lambda$, so computing the sensitivity adds almost no overhead beyond a pointwise multiplication and absolute value.

To reduce noise from stochastic gradients, AdaLoRA maintains an exponential moving average of the sensitivity estimates across training steps:

$$\bar{s}_i^{(t)} = \beta \cdot \bar{s}_i^{(t-1)} + (1 - \beta) \cdot s_i^{(t)}$$

where:

• $\bar{s}_i^{(t)}$: the smoothed sensitivity estimate at step $t$, averaging information over the training history
• $\beta$: the smoothing parameter, typically set around 0.85, controlling how much weight to give the historical average versus the new observation
• $s_i^{(t)}$: the instantaneous sensitivity from the current step

This averaging stabilizes importance estimates across mini-batches, preventing premature pruning decisions based on noisy gradient samples. Because gradients computed on small mini-batches can vary materially from step to step, a single noisy gradient could incorrectly suggest that an useful component is unimportant, or vice versa. The exponential moving average smooths out this noise by giving weight to the entire history of gradient observations, with more recent observations weighted more heavily through the $\beta$ decay factor. The parameter $\beta$ controls the effective window: with $\beta = 0.85$, roughly $\frac{1}{1-\beta} = 6.7$ recent steps receive large weight, while older steps fade away exponentially. Higher values of $\beta$ create smoother estimates that respond more slowly to real changes in importance; lower values track changes faster but are more susceptible to noise.

When the orthonormality constraint is enforced only softly, the actual magnitude of each component's contribution to the weight update is not determined solely by $\lambda_i$ but also depends on the norms of $P_{:,i}$ and $Q_{i,:}$. In the ideal SVD case where $\|P_{:,i}\|_2 = 1$ and $\|Q_{i,:}\|_2 = 1$, the singular value alone captures the full magnitude. In practice, slight deviations from orthonormality mean that the vectors may have norms slightly different from one, which affect the actual magnitude of the corresponding rank-1 contribution.

To account for this, and to ensure that importance scores are fairly comparable across weight matrices with different dimensions, AdaLoRA uses a complete importance formula that incorporates the vector norms:

$$S_i^{(t)} = \bar{s}_i^{(t)} \cdot |\lambda_i^{(t)}| \cdot \frac{\|P_{:,i}\|_F \cdot \|Q_{i,:}\|_F}{\sqrt{d} \cdot \sqrt{d_{\text{in}}}}$$

where:

• $S_i^{(t)}$: the final importance score used for global ranking at step $t$
• $\bar{s}_i^{(t)}$: the smoothed sensitivity estimate from the exponential moving average
• $|\lambda_i^{(t)}|$: the absolute magnitude of the current singular value
• $\|P_{:,i}\|_F$: the Frobenius norm of the $i$-th column of $P$, correcting for column-length deviations from unit norm
• $\|Q_{i,:}\|_F$: the Frobenius norm of the $i$-th row of $Q$, correcting for row-length deviations from unit norm
• $d, d_{\text{in}}$: the output and input dimensions of the weight matrix, used for cross-layer normalization

The normalization by $\sqrt{d}$ and $\sqrt{d_{\text{in}}}$ is needed for fair comparison across layers. Without this normalization, layers with larger dimensions would naturally have larger vector norms (since the column of $P$ has $d$ entries and thus a norm that scales roughly as $\sqrt{d}$ for random initialization), potentially biasing the pruning toward keeping components in smaller layers regardless of their actual importance for the task.

When the orthonormality constraint is well-satisfied, $\|P_{:,i}\|_F \approx 1$ and $\|Q_{i,:}\|_F \approx 1$, and the full formula simplifies to approximately $\bar{s}_i^{(t)} \cdot |\lambda_i^{(t)}|$, recovering the intuitive product of sensitivity and magnitude. The full formula is thus a reliable version of this product that remains valid even when orthonormality is imperfectly enforced, which is always the case in practice.