Pruning Basics › Pruning Basics

Modern language models are extraordinarily large. GPT-2 has 1.5 billion parameters; GPT-3 has 175 billion; the largest production models today exceed hundreds of billions. Deploying these models on mobile devices, embedded hardware, or cost-sensitive cloud infrastructure is impractical without aggressive size reduction. One of the oldest and most principled techniques for shrinking neural networks is pruning: the systematic removal of weights, neurons, or entire layers that contribute little to the model's output.

The core insight behind pruning is that trained neural networks are highly redundant. Not every parameter is equally important. Some weights are large and actively shape predictions; others hover near zero and could be removed with little loss of accuracy. Pruning exploits this redundancy by identifying the least important parameters and eliminating them, leaving a smaller, faster, cheaper model behind.

Pruning has a long history in neural network research. The foundational ideas appeared in the early 1990s with Optimal Brain Damage (LeCun et al., 1990) and Optimal Brain Surgeon (Hassibi et al., 1993), which used second-order information to decide which weights to remove from small convolutional networks. The technique was largely overshadowed during the 2000s when support vector machines and other methods dominated, but it returned to prominence with the revival of deep learning. Han et al.'s 2015 paper "Learning both Weights and Connections for Efficient Neural Networks" demonstrated that modern deep networks could be pruned to 90% sparsity with negligible accuracy loss, and the field has grown rapidly since then. Today, pruning is a standard part of the model compression toolkit alongside quantization and knowledge distillation.

In this chapter, we focus on the fundamentals: what pruning is, how to decide what to remove, what granularity of removal to apply, and how to schedule the removal over time. The next chapter on Structured Pruning dives deeper into head pruning, layer pruning, and width pruning for transformer architectures specifically. Here, we build the foundation that makes those advanced techniques comprehensible.

Pruning sits alongside knowledge distillation, which we covered in the previous two chapters, and quantization as the primary model compression toolkit. Where distillation trains a small model to mimic a large one, and quantization reduces the numerical precision of existing weights, pruning removes weights entirely. Each technique attacks model size from a different angle, and practitioners often combine all three for maximum compression.