Structured Pruning › Summary

Structured pruning removes entire architectural components from transformers, producing models that run faster on any hardware without requiring specialized sparse computation support.

• Head pruning removes entire attention heads using importance scores based on gradient magnitudes, activation norms, or loss sensitivity. After removal, the attention projection matrices shrink and every forward pass uses fewer FLOPs. Global pruning across all layers outperforms layer-by-layer pruning because it identifies layers with more redundant heads.
• Layer pruning removes complete transformer blocks. The cosine similarity between a block's input and output identifies near-identity layers that contribute little. Middle layers are often the best candidates, following the U-shaped importance profile observed across BERT, GPT, and Llama model families. Angular distance (via $\arccos$) provides a scale-invariant version of this criterion.
• Width pruning reduces the internal dimensions of feedforward sublayers or attention head dimensions by eliminating low-activation neurons. This offers fine-grained control over model size that complements the coarser-grained head and layer pruning strategies. Gated feedforward layers (SwiGLU) require coordinated pruning of both the gate and value projections.
• Iterative pruning with recovery fine-tuning between rounds achieves better final quality than a single aggressive pruning pass. Cubic or polynomial schedules balance compression speed and quality retention by allowing the model more adaptation time at the critical early and late stages of the pruning process.
• Calibration data quality determines how accurate importance scores are. Use representative data from the actual deployment context, and keep calibration and fine-tuning data separate to avoid overfitting.
• Combining techniques amplifies compression ratios. Structured pruning reduces architecture size, quantization reduces numerical precision, and distillation transfers knowledge from a larger model. Together, these techniques make large language models deployable in contexts where the original models would be impractical.

Structured pruning is most powerful when combined with other compression techniques. Pruning reduces architecture size, quantization reduces numerical precision, and distillation transfers knowledge from a larger model. Together, these techniques make large language models deployable in contexts where the original models would be impractical. The next part of this handbook explores reasoning capabilities in language models, where we will see how efficient inference from compressed models enables deploying chain-of-thought reasoning in constrained environments.