Pruning Basics › What to Prune: Pruning Criteria

A pruning criterion is a function that assigns an importance score to each element being considered for removal. Elements with low importance scores are pruned first. The choice of criterion has a large effect on how gracefully a model degrades as sparsity increases, and choosing the right criterion for your use case is as important as choosing the right schedule.

Several pruning criteria are commonly used in practice, ranging from the extremely simple to the computationally intensive.

The simplest and most widely used criterion is weight magnitude. The assumption is direct: weights close to zero contribute little to the network's output and can be safely removed.

Formally, for a weight $w_{ij}$ connecting neuron $i$ in layer $l$ to neuron $j$ in layer $l+1$, the importance score is:

$$s_{ij} = |w_{ij}|$$

where:

• $w_{ij}$: the scalar weight value connecting neurons $i$ and $j$
• $s_{ij}$: the importance score assigned to that weight, where higher scores indicate greater importance

Weights are sorted by importance score in ascending order. The bottom $k\%$ by score, where $k$ is the target sparsity ratio, are pruned by setting them to zero.

The simplicity of magnitude pruning belies its effectiveness. Despite requiring no gradient computation and no forward pass over a dataset, magnitude pruning competes well with more sophisticated methods on most standard benchmarks. Han et al.'s original 2015 results pruning LeNet and AlexNet used magnitude pruning almost exclusively. BERT pruning experiments consistently find that magnitude-based methods achieve accuracy within one to two points of gradient-based methods at equivalent sparsity levels, at a fraction of the computational cost.

Why does such a simple heuristic work so well? The answer is that gradient descent implicitly drives unimportant weights toward zero. When a weight is irrelevant to the loss, the gradient pushes it in directions that do not systematically increase its magnitude, and weight decay (L2 regularization) actively shrinks it. By the end of training, weights that the network does not use have been nudged toward zero. Magnitude pruning is, in a sense, reading the signal that the optimization process has already embedded in the weight values.

Magnitude pruning has a meaningful weakness: it ignores context. A small weight might contribute significantly if it lies on an information-critical path through the network. A neuron that connects two important computation chains might have small incoming weights because the downstream layer has large weights that amplify its output. Removing the small upstream weight then indirectly damages the large downstream computation. Magnitude pruning cannot detect this kind of dependency. Gradient-based and second-order methods are designed to address exactly this limitation.

Another subtlety is the choice between local and global magnitude pruning. Local pruning applies a sparsity threshold within each layer independently, making sure each layer reaches the same sparsity ratio. Global pruning ranks all weights in the network together, regardless of layer, and prunes the bottom fraction globally. In practice, global pruning typically produces better results because some layers are more critical than others. The softmax layer and the embedding layer may be very sensitive to pruning, and global pruning naturally prunes them less aggressively than layers with more redundancy.

A more detailed criterion uses gradient information alongside weight magnitude. The intuition is that the gradient of the loss with respect to a weight tells us how much the loss would change if that weight changed slightly. A weight with a large gradient magnitude is currently being updated aggressively and is likely doing important work; a weight with a near-zero gradient is not contributing to loss reduction and may be dormant.

One concrete formulation is the gradient-magnitude product, sometimes called the saliency score:

$$s_{ij} = \left| w_{ij} \cdot \frac{\partial \mathcal{L}}{\partial w_{ij}} \right|$$

where:

• $w_{ij}$: the current weight value
• $\frac{\partial \mathcal{L}}{\partial w_{ij}}$: the partial derivative of the loss $\mathcal{L}$ with respect to $w_{ij}$
• $s_{ij}$: the importance score, capturing both the current size and the sensitivity of the weight

This product has a natural interpretation via a first-order Taylor expansion of the loss. If we were to set weight $w_{ij}$ to zero (a change of $-w_{ij}$ from its current value), the change in loss would be approximately:

$$\Delta \mathcal{L} \approx \frac{\partial \mathcal{L}}{\partial w_{ij}} \cdot (-w_{ij}) = -w_{ij} \cdot \frac{\partial \mathcal{L}}{\partial w_{ij}}$$

The absolute value of this quantity, $|w_{ij} \cdot \partial \mathcal{L}/\partial w_{ij}|$, estimates how much zeroing the weight would change the loss. A small product means removing the weight has little impact on the loss. This is a stronger criterion than magnitude alone because it accounts for gradient context: a small weight that sits at a high-gradient position is treated as more important than a small weight at a flat region.

The gradient-magnitude product requires one forward pass and one backward pass over a calibration dataset to compute the gradients. This is more expensive than pure magnitude pruning but still tractable for most models. The gradient values change during fine-tuning, so this criterion is typically computed once on a representative batch before each pruning step.

One subtlety is that gradient values can be noisy, especially in mini-batch training. Using gradients averaged over multiple batches or an entire epoch reduces noise and gives more reliable importance estimates. Some implementations accumulate the gradient signal over a window of training steps rather than using a single batch.

A theoretically stronger criterion uses second-order information: the curvature of the loss field. The main insight is that the first-order Taylor approximation ignores the curvature of the loss, which can be significant. Two weights with identical gradient-magnitude products might have very different actual impacts on the loss if the loss field curves steeply at one location and gently at the other.

The Optimal Brain Damage (OBD) method, published by LeCun, Denker, and Solla in 1990, introduced the use of the Hessian for weight importance. OBD derives saliency scores by computing the second-order change in loss when each weight is set to zero. Under the assumption that the Hessian is diagonal (each weight's effect is independent), the OBD saliency score for weight $w_{ij}$ is:

$$s_{ij} = \frac{1}{2} \cdot \frac{\left(\frac{\partial \mathcal{L}}{\partial w_{ij}}\right)^2}{H_{ij,ij}}$$

where:

• $\frac{\partial \mathcal{L}}{\partial w_{ij}}$: the gradient of the loss with respect to $w_{ij}$
• $H_{ij,ij}$: the diagonal element of the Hessian matrix corresponding to $w_{ij}$, the second derivative of the loss with respect to $w_{ij}$ (also called the curvature)
• $s_{ij}$: the estimated increase in loss from zeroing weight $w_{ij}$

The denominator $H_{ij,ij}$ acts as a normalizer. A weight with a large gradient but flat curvature is riskier to prune than a weight with the same gradient but steep curvature, because in the steep case, nearby weights can more easily compensate via fine-tuning. The OBD formula captures this intuition.

The successor method, Optimal Brain Surgeon (OBS), removes the diagonal approximation and accounts for off-diagonal Hessian terms. OBS also derives the optimal weight adjustment for the remaining weights after each pruning step, so accuracy loss is minimized at each removal. OBS is more accurate than OBD but requires solving a linear system involving the full Hessian inverse, which is intractable for networks with millions of parameters. Various approximations have been developed to make OBS tractable at scale.

Modern methods like SparseGPT (Frantar and Alistarh, 2023) apply OBS-style ideas to prune large language models in a single pass without any fine-tuning, using efficient block-wise Hessian approximations that scale to billions of parameters. SparseGPT can prune GPT-3 variants to 50% sparsity with less than one percent increase in perplexity. This demonstrates that second-order methods, when carefully implemented, substantially outperform magnitude-based methods at high sparsity for large models.

The computational cost of second-order methods is their main barrier. Computing and storing the Hessian for a modern language model is infeasible without approximations. Diagonal Hessian estimates (from Fisher information or empirical risk curvature) are tractable but imprecise. Block-wise approaches that compute the Hessian for small groups of layers independently are a practical middle ground. As hardware and software improve, expect second-order pruning methods to become more common in production pipelines.

For neurons or attention heads rather than individual weights, a natural criterion is the magnitude of the neuron's activation across a representative dataset. A neuron that consistently produces near-zero activations regardless of the input is not contributing meaningful computation and can be safely removed.

Given a calibration dataset $\mathcal{D}$ of $N$ examples, the importance score for neuron $j$ is:

$$s_j = \frac{1}{N} \sum_{x \in \mathcal{D}} |a_j(x)|$$

where:

• $a_j(x)$: the activation of neuron $j$ on input $x$
• $N$: the number of examples in the calibration dataset
• $s_j$: the mean absolute activation, averaged over the dataset

This criterion does not require gradients and is cheap to compute: a single forward pass over the calibration data gives activations for all neurons. The calibration dataset should be representative of the model's deployment distribution; using an unrepresentative set can lead to removing neurons that are important for common inputs but happen to fire rarely on the calibration examples.

Activation-based criteria are particularly useful when pruning at a coarser granularity than individual weights, such as pruning entire attention heads or feed-forward neurons in a transformer. For attention head pruning, the activation criterion can be adapted to measure the average entropy of the head's attention distribution or the variance of its outputs across inputs, both of which capture how actively the head is being used.

The four criteria we have covered represent a spectrum from simple to sophisticated. In practice, the choice depends on the available compute budget and the target sparsity level.

At low to moderate sparsity (below 50%), magnitude pruning often achieves near-optimal results at minimal cost. The gradient information encoded in the weight values is sufficient to identify clearly unimportant parameters. At high sparsity (above 70%), gradient-based and second-order methods show more significant advantages because the decision of which weights to keep becomes more consequential. At 90% sparsity, removing the wrong 9% can cause much larger accuracy drops than at 50% sparsity.

For structured pruning of transformer components (heads, neurons, layers), activation-based criteria are often preferred because they directly measure whether a structural unit is doing useful computation, rather than measuring individual weight importance within the unit.