Structured Pruning › Pruning Schedule and Iterative Pruning

A single aggressive pruning pass often damages quality more than a series of smaller pruning steps with fine-tuning between each step. Iterative pruning cycles through: prune a small amount, fine-tune to recover, evaluate quality, and repeat until the target size is reached. Each fine-tuning phase allows the model to stabilize before the next round of pruning begins. The accumulated effect of many small, recoverable pruning steps is a final model that is both smaller and higher quality than you would get from one large pruning step.

The intuition behind iterative pruning is that importance scores change after each pruning round. When you remove a low-importance head, the remaining heads may redistribute their roles. A head that appeared unimportant under the original model may become more important when another head in the same layer is removed, because the remaining head must now cover both roles. Iterative pruning updates importance scores between rounds, catching these redistributions and making better decisions.

The pruning schedule can be described as a function that determines what fraction of the model to prune at each step $t$:

$$s_t = s_f + (s_i - s_f) \cdot \left(1 - \frac{t - t_0}{n \cdot \Delta t}\right)^3$$

where:

• $s_t$ is the sparsity at step $t$
• $s_i$ is the initial sparsity (often 0)
• $s_f$ is the final target sparsity
• $t_0$ is the step at which pruning begins
• $n$ is the number of pruning steps
• $\Delta t$ is the interval between pruning steps

The cubic term $\left(1 - \frac{t - t_0}{n \cdot \Delta t}\right)^3$ is the key feature of this schedule. At the beginning of pruning (when $t$ is just above $t_0$), the term is close to 1, so $s_t$ is close to $s_i$: pruning starts slowly. As $t$ increases toward $t_0 + n \cdot \Delta t$, the term approaches 0 and $s_t$ approaches $s_f$: pruning accelerates toward the target. Near the end of the schedule, as the target approaches, the cubic function causes pruning to decelerate again, making fine adjustments rather than large jumps. This S-shaped progression (slow, fast, slow) is better matched to how models learn than a constant linear rate.

The cubic schedule starts pruning slowly, accelerates through the middle, then slows down again near the target. This allows the model to accommodate each wave of removals before more components are pruned. The deceleration near the target is particularly important: the final few percentage points of sparsity are often the hardest to achieve without quality loss, and the schedule naturally allocates more fine-tuning time near the target.

The chart below compares the cubic schedule to a linear schedule. The cubic schedule removes fewer components early in training, when the model is still adjusting from prior fine-tuning, and reaches the target sparsity more gradually.