AdaLoRA › Worked Example: Step-by-Step AdaLoRA Pruning

To build concrete intuition for how AdaLoRA's pruning decisions unfold, let's work through a small numerical example from first principles. We will consider a toy model with just two adapted weight matrices, each initialized at rank 3, with a target budget of 3 total triplets (meaning we need to prune half the components).

Setup: Two weight matrices $W_1$ and $W_2$, each with initial rank $r_0 = 3$. Total initial budget: $2 \times 3 = 6$ triplets. Target budget: $b = 3$ triplets. We are at some training step after the warmup period.

Step 1: Read current singular values and gradients.

For matrix $W_1$, suppose the current singular values and their gradients are:

• Triplet $(1,1)$: $\lambda_1 = 0.80$, $\frac{\partial \mathcal{L}}{\partial \lambda_1} = -0.20$
• Triplet $(1,2)$: $\lambda_2 = 0.45$, $\frac{\partial \mathcal{L}}{\partial \lambda_2} = -0.60$
• Triplet $(1,3)$: $\lambda_3 = 0.10$, $\frac{\partial \mathcal{L}}{\partial \lambda_3} = -0.05$

For matrix $W_2$:

• Triplet $(2,1)$: $\lambda_4 = 0.70$, $\frac{\partial \mathcal{L}}{\partial \lambda_4} = -0.15$
• Triplet $(2,2)$: $\lambda_5 = 0.35$, $\frac{\partial \mathcal{L}}{\partial \lambda_5} = -0.40$
• Triplet $(2,3)$: $\lambda_6 = 0.05$, $\frac{\partial \mathcal{L}}{\partial \lambda_6} = -0.02$

Step 2: Compute instantaneous sensitivities.

$$s_i^{(t)} = \left| \lambda_i \cdot \frac{\partial \mathcal{L}}{\partial \lambda_i} \right|$$

For $W_1$:

• $s_1 = |0.80 \times (-0.20)| = |-0.160| = 0.160$
• $s_2 = |0.45 \times (-0.60)| = |-0.270| = 0.270$
• $s_3 = |0.10 \times (-0.05)| = |-0.005| = 0.005$

For $W_2$:

• $s_4 = |0.70 \times (-0.15)| = |-0.105| = 0.105$
• $s_5 = |0.35 \times (-0.40)| = |-0.140| = 0.140$
• $s_6 = |0.05 \times (-0.02)| = |-0.001| = 0.001$

Notice that triplet $(1,2)$ (the second component of $W_1$) has the highest sensitivity despite not having the largest magnitude. This is because its gradient is large in magnitude ($-0.60$), showing the loss is sensitive to this component's value. Triplet $(1,1)$ has a larger magnitude but smaller gradient, suggesting it has already settled near a good value.

Step 3: Update exponential moving averages (assuming previous smoothed estimates).

Suppose the previous smoothed estimates were $\bar{s}_i^{(t-1)} = [0.15, 0.25, 0.01, 0.10, 0.13, 0.002]$ and $\beta = 0.85$:

$$\bar{s}_i^{(t)} = 0.85 \cdot \bar{s}_i^{(t-1)} + 0.15 \cdot s_i^{(t)}$$

For $W_1$:

• $\bar{s}_1 = 0.85 \times 0.15 + 0.15 \times 0.160 = 0.1275 + 0.024 = 0.1515$
• $\bar{s}_2 = 0.85 \times 0.25 + 0.15 \times 0.270 = 0.2125 + 0.0405 = 0.2530$
• $\bar{s}_3 = 0.85 \times 0.01 + 0.15 \times 0.005 = 0.0085 + 0.00075 = 0.0093$

For $W_2$:

• $\bar{s}_4 = 0.85 \times 0.10 + 0.15 \times 0.105 = 0.085 + 0.01575 = 0.1008$
• $\bar{s}_5 = 0.85 \times 0.13 + 0.15 \times 0.140 = 0.1105 + 0.021 = 0.1315$
• $\bar{s}_6 = 0.85 \times 0.002 + 0.15 \times 0.001 = 0.0017 + 0.00015 = 0.0019$

Step 4: Compute final importance scores. For simplicity, assume the orthonormality constraint is well-satisfied so that $\|P_{:,i}\|_F \approx 1$ and $\|Q_{i,:}\|_F \approx 1$, and assume all layers have the same dimensions so the normalization factor is the same for all. The importance score simplifies to $S_i = \bar{s}_i \cdot |\lambda_i|$.

• $S_1 = 0.1515 \times 0.80 = 0.1212$
• $S_2 = 0.2530 \times 0.45 = 0.1139$
• $S_3 = 0.0093 \times 0.10 = 0.0009$
• $S_4 = 0.1008 \times 0.70 = 0.0706$
• $S_5 = 0.1315 \times 0.35 = 0.0460$
• $S_6 = 0.0019 \times 0.05 = 0.0001$

Step 5: Global ranking and pruning. Sorting all six triplets by importance score in descending order:

1. Triplet $(1,1)$: $S = 0.1212$ (keep)
2. Triplet $(1,2)$: $S = 0.1139$ (keep)
3. Triplet $(2,1)$: $S = 0.0706$ (keep)
4. Triplet $(2,2)$: $S = 0.0460$ (prune)
5. Triplet $(1,3)$: $S = 0.0009$ (prune)
6. Triplet $(2,3)$: $S = 0.0001$ (prune)

With a budget of 3, we keep triplets 1, 2, and 4 (the top 3 by importance) and mask the rest by setting their $\lambda$ values to zero. The resulting effective rank distribution is: $W_1$ gets rank 2 (keeps triplets 1 and 2), $W_2$ gets rank 1 (keeps triplet 4 only).

Step 6: Interpretation. This allocation shows the data-driven discovery that $W_1$ is more important for this task than $W_2$: it receives twice the parameter budget. Within $W_1$, the first two triplets both have importance scores above 0.11 and are kept, while the third has an importance score near zero and is discarded. Within $W_2$, only the first triplet has a real importance score; the second and third are pruned.

Notice something interesting about the pruning decision for triplet $(2,2)$ versus $(1,3)$: triplet $(2,2)$ has a much larger magnitude ($|\lambda| = 0.35$ versus $|\lambda| = 0.10$) but is still pruned, while triplet $(1,3)$ is also pruned. What matters is the combined importance score: even $W_1$'s smallest triplet ranks 5th overall with $S = 0.0009$, just barely above triplet $(2,3)$. A layer-by-layer approach that independently pruned within each matrix might make a different decision, but the global competition yields an allocation where total budgeted importance is maximized.