Pruning Basics › Summary

Pruning removes weights, neurons, or structural components from trained neural networks by exploiting the redundancy introduced during overparameterized training. The history of the technique spans from LeCun's Optimal Brain Damage in 1990 through the lottery ticket hypothesis and modern one-shot methods for billion-parameter language models.

The key concepts covered in this chapter are:

• Why pruning works: Trained networks are heavily overparameterized. The lottery ticket hypothesis provides a formal account: every large network contains a smaller subnetwork that can match the full network's accuracy when trained from the same initialization. Pruning finds this subnetwork after the fact.
• Pruning criteria determine which weights are removed. Magnitude pruning (removing the smallest weights by absolute value) is the simplest and most widely used. Gradient-based methods use the saliency score $|w \cdot \partial \mathcal{L} / \partial w|$ for more principled removal. Second-order methods using Hessian information (OBD, OBS, SparseGPT) make more accurate predictions about which weights are safe to remove, especially at high sparsity.
• Unstructured pruning removes individual weights at arbitrary positions, achieving high compression with minimal accuracy loss but requiring sparse compute support for real hardware speedups.
• Structured pruning removes entire neurons, heads, or layers, producing a smaller dense network that runs efficiently on standard hardware at the cost of higher accuracy loss per unit of compression.
• Semi-structured (N:M) pruning provides a middle ground: regular sparsity patterns that NVIDIA Ampere hardware can accelerate natively, giving measurable speedups while allowing finer-grained removal than fully structured pruning.
• Pruning schedules control the trajectory from dense to sparse. One-shot pruning applies the full reduction at once; iterative pruning alternates prune-and-finetune cycles; gradual pruning increases sparsity continuously during training. Iterative and gradual schedules consistently outperform one-shot pruning at high sparsity. Weight rewinding, motivated by the lottery ticket hypothesis, resets surviving weights to early-training checkpoints for potentially better subnetwork training.
• Fine-tuning after pruning is essential for recovering accuracy. The pruned model adjusts the magnitudes of surviving weights to compensate for the removed connections.
• The sparsity-efficiency gap: Achieving real hardware speedups from sparsity requires either structured pruning, N:M sparsity on supporting hardware, or sparse compute infrastructure. Unstructured pruning on standard hardware primarily reduces model size, not inference latency.

The next chapter on Structured Pruning applies these fundamentals specifically to transformer architectures, examining how to prune attention heads, feed-forward neurons, and entire layers while preserving as much of the model's language understanding as possible.