Pruning Basics › Why Pruning Works

Before diving into how pruning works, it is worth understanding why it works at all. Why should it be possible to remove a large fraction of weights from a trained network without significant accuracy loss?

The answer lies in two related phenomena: overparameterization and the lottery ticket hypothesis. Both provide complementary explanations for the same empirical observation: trained networks contain enormous amounts of redundancy.

Neural networks are routinely trained with far more parameters than are theoretically necessary to represent the solution. A binary text classifier on a 100-dimensional feature space does not need millions of parameters to learn a decision boundary. A language model trained on the English Wikipedia does not need 175 billion parameters to represent the statistical regularities of English text. Yet we train these oversized networks, and they work extraordinarily well.

This overparameterization is not accidental; it serves a essential purpose during training. Gradient descent on a highly overparameterized network is much easier to optimize than on a minimal network. The loss field of an overparameterized model has fewer sharp local minima and saddle points, and random initialization is far more likely to land in a region from which gradient descent can find a good solution. The gradient signal is also more reliable: each parameter update has a small relative impact on the overall loss, making optimization stable and predictable. In compact, under-parameterized networks, each weight carries more responsibility, the loss field is sharper, and optimization is more sensitive to initialization and learning rate.

Once training is complete, however, the extra capacity is no longer needed. The network has converged to a solution that uses only a fraction of its theoretical expressiveness. The surplus parameters are effectively frozen echoes of the training dynamics, not active contributors to the learned function. They can be removed without meaningful degradation because the surviving parameters already encode everything the model needs.

Empirical evidence strongly supports this view. Researchers have pruned 90 to 99 percent of parameters from trained image classifiers and language models while retaining nearly all predictive accuracy. The BERT model, with 110 million parameters, can be pruned to 40 percent sparsity with less than one point of accuracy loss on GLUE benchmarks. GPT-2 shows similar resilience. The trained network is not merely large; it is the compressed solution wrapped in an enormous amount of structural redundancy.

The redundancy shows up in several forms. Many weights in a trained network are near-zero, contributing negligible output to any downstream computation. Other weights are highly correlated with nearby weights in the same layer, meaning multiple weights encode nearly identical information. Some neurons in fully-connected layers fire rarely, activating only on a small fraction of inputs, and removing them affects only those rare cases. All of these patterns are exploitable by pruning.

A more formal account of why pruning works comes from the Lottery Ticket Hypothesis, proposed by Frankle and Carlin in 2019. The hypothesis states: a randomly initialized dense network contains a subnetwork (a "winning ticket") that, when trained in isolation from the same initial weights, can match the accuracy of the full network in comparable training time.

The key phrase is "from the same initial weights." The hypothesis is not simply that a small subnetwork exists within the full network after training. It is that there exists a specific subnetwork whose initial weights, drawn from the same random initialization as the full network, are a good starting point for learning. The initial random values of the winning ticket's weights matter. If you take the winning ticket's architecture but re-initialize its weights randomly, it may not train as well.

The lottery ticket metaphor is apt. When you buy a lottery ticket, you do not know at purchase time which ticket will win. Similarly, when you randomly initialize a large network, the winning ticket is hidden inside it, but you cannot identify it in advance. You have to run the full training process, then look backward to find which subnetwork was doing the real work.

The practical implications are significant. First, it explains why pruning works: you are essentially discovering the winning ticket after the fact. Second, it motivates a pruning methodology: train the full network, prune to find the winning ticket structure, then potentially retrain that structure from the original initialization (a strategy called weight rewinding, which we cover later in the chapter). Third, it suggests that network architecture is not the only thing that matters; the specific pattern of which weights are nonzero is itself a form of architecture.

The hypothesis also has notable limitations. Frankle and Carlin's original experiments worked well on small networks. Scaling the approach to large transformer models revealed complications: the winning ticket at initialization is harder to find for very deep networks, and the original strong form of the hypothesis (rewinding to step 0) sometimes fails, though a weaker form (rewinding to an early-training checkpoint rather than initialization) tends to work. We revisit this in the pruning schedules section.

Transformer-based language models exhibit a particularly interesting form of redundancy. Attention heads within the same layer often compute related or partially redundant functions. In multi-head self-attention, each head attends to different positions, but heads within a layer frequently specialize in similar syntactic or positional patterns, and removing several heads from the same layer often has minimal impact on performance. Michel et al. (2019) demonstrated that BERT retains competitive performance after removing 50 to 60 percent of attention heads from most layers. This suggests that language-model redundancy appears throughout the architecture and is especially concentrated in the attention mechanism.

Feed-forward layers in transformers also contain substantial redundancy. The intermediate dimension of a transformer's feed-forward sublayer is typically four times the model dimension, a ratio that comes from the original transformer paper without a strong theoretical justification. Subsequent work has shown that this 4x ratio is conservative; models trained with smaller feed-forward dimensions or with a fraction of the feed-forward neurons removed retain most of their capability.

Understanding these structural patterns is what distinguishes naive magnitude pruning from principled structured pruning for transformers. The fundamentals covered in this chapter apply uniformly; the next chapter specializes them to the transformer architecture.