Structured Pruning › Limitations and Practical Considerations

Structured pruning delivers real speedups but comes with significant practical challenges that affect how aggressively you can prune in production settings.

The most fundamental limitation is the accuracy-efficiency tradeoff. Every component you remove carries some information that the model was using, even if the component was relatively unimportant. With mild pruning (removing 10-20% of heads or 1-2 layers from a 12-layer model), quality usually recovers fully after fine-tuning. With aggressive pruning (removing 40-50% of components), some permanent quality degradation is unavoidable. The degree of degradation depends on the task: tasks that require broad linguistic knowledge degrade more than narrow tasks that use only a subset of the model's capabilities.

Recovery fine-tuning requires labeled data for the downstream task. This is straightforward for supervised classification tasks but harder for generative models, where you need a proxy objective that correlates with generation quality. Using next-token prediction loss on a representative text corpus works for language models, but the quality of the pruned generative model may not improve as smoothly as for classifiers. For generative models, human evaluation or automatic metrics like ROUGE or BERTScore should be used to track quality throughout the pruning process, not just training loss.

The importance score computation itself has limitations. Gradient-based importance scores are computed on a specific calibration dataset and may not generalize across domains. A head that is unimportant for formal news text may be necessary for informal conversational text. This means pruning decisions made for one deployment context may not transfer well to other contexts, requiring re-pruning when the use case changes.

Structured pruning interacts with other compression techniques in important ways. Applying quantization after pruning compounds the compression but also compounds potential quality loss. Applying distillation after pruning is more stable: you prune first to get the target architecture, then distill to recover quality from a larger teacher. This two-stage pipeline (prune then distill) often outperforms either technique applied alone.

Head pruning has an interesting failure mode: when you remove an attention head, the surviving heads may not fully compensate because their query and key spaces were trained independently. Multi-head attention does not guarantee that any single head can replicate the behavior of another. Recovery fine-tuning helps, but there are limits to how much compensation is possible without retraining from scratch.

Layer pruning has a different failure mode: removing middle layers can disrupt the flow of information from early to late layers. If removed layers were responsible for building up specific intermediate representations that later layers relied on, the resulting model cannot route information correctly even after fine-tuning. Careful importance scoring based on layer similarity helps identify which layers are safe to remove, but abrupt removal of critical layers can be difficult to recover from. This is why the iterative approach (remove one layer at a time with fine-tuning between rounds) is more reliable than removing multiple layers simultaneously.

Finally, structured pruning is sensitive to the model's pretraining history. Models that were pretrained with larger batches and for more steps tend to be more amenable to pruning, because they have had more opportunity to develop redundancy in their components. Models that were undertrained relative to their size (common in resource-constrained academic settings) often cannot be pruned as aggressively because their components are not as redundant. When working with a model of unknown provenance, start with a conservative pruning ratio and increase gradually while monitoring quality.