Structured Pruning › Structured Pruning

Modern language models have grown to sizes where deploying them in production is difficult. A model with billions of parameters demands significant memory, processing time, and energy on every inference call. Knowledge distillation, which we covered earlier in this part, trains a smaller model to mimic a larger one. Quantization reduces the numerical precision of weights. Pruning takes a different approach: it removes parts of an existing model entirely, creating a physically smaller architecture that performs nearly as well as the original.

Pruning broadly divides into two families. Unstructured pruning, covered in the previous chapter, sets individual weights to zero without changing the model's computational graph. The weight matrices remain the same shape, which means standard GPU kernels see no speedup unless you use specialized sparse computation hardware. Structured pruning, the focus of this chapter, removes entire structural components: whole attention heads, entire transformer layers, or full rows and columns from weight matrices. Because the model architecture changes shape, structured pruning produces real speedups on any hardware without special sparse kernels.

The distinction matters enormously in practice. When you remove a single attention head from a 12-head transformer layer, the resulting layer has 11 heads and its weight matrices shrink proportionally. The forward pass takes less memory and fewer FLOPs on every call. When you remove a full transformer layer from a 12-layer model, every inference call skips that layer entirely. These are concrete, measurable speedups that show up in production latency metrics. Structured pruning is not about theoretical sparsity; it is about real hardware-visible compression that you can benchmark directly.

The field of structured pruning for transformers emerged from broader work on network pruning in convolutional networks, where researchers found that removing entire filters rather than individual weights produced better hardware acceleration. When BERT arrived in 2018 and researchers began analyzing its attention patterns, it quickly became clear that the same principle applied: many heads were redundant, many layers were near-identity, and removing them in bulk produced models that were both smaller and faster without catastrophic quality loss. This observation motivated a wave of papers on head pruning and layer dropping that established the foundational techniques we will study here.