Pruning: Structured and Unstructured Methods

Modern language models are extraordinarily large. GPT-2 has 1.5 billion parameters; GPT-3 has 175 billion; the largest production models today exceed hundreds of billions. Deploying these models on mobile devices, embedded hardware, or cost-sensitive cloud infrastructure is impractical without aggressive size reduction. One of the oldest and most principled techniques for shrinking neural networks is pruning: the systematic removal of weights, neurons, or entire layers that contribute little to the model's output.

The core insight behind pruning is that trained neural networks are highly redundant. Not every parameter is equally important. Some weights are large and actively shape predictions; others hover near zero and could be removed with little loss of accuracy. Pruning exploits this redundancy by identifying the least important parameters and eliminating them, leaving a smaller, faster, cheaper model behind.

Pruning has a long history in neural network research. The foundational ideas appeared in the early 1990s with Optimal Brain Damage (LeCun et al., 1990) and Optimal Brain Surgeon (Hassibi et al., 1993), which used second-order information to decide which weights to remove from small convolutional networks. The technique was largely overshadowed during the 2000s when support vector machines and other methods dominated, but it returned to prominence with the revival of deep learning. Han et al.'s 2015 paper "Learning both Weights and Connections for Efficient Neural Networks" demonstrated that modern deep networks could be pruned to 90% sparsity with negligible accuracy loss, and the field has grown rapidly since then. Today, pruning is a standard part of the model compression toolkit alongside quantization and knowledge distillation.

In this chapter, we focus on the fundamentals: what pruning is, how to decide what to remove, what granularity of removal to apply, and how to schedule the removal over time. The next chapter on Structured Pruning dives deeper into head pruning, layer pruning, and width pruning for transformer architectures specifically. Here, we build the foundation that makes those advanced techniques comprehensible.

Pruning sits alongside knowledge distillation, which we covered in the previous two chapters, and quantization as the primary model compression toolkit. Where distillation trains a small model to mimic a large one, and quantization reduces the numerical precision of existing weights, pruning removes weights entirely. Each technique attacks model size from a different angle, and practitioners often combine all three for maximum compression.

Before diving into how pruning works, it is worth understanding why it works at all. Why should it be possible to remove a large fraction of weights from a trained network without significant accuracy loss?

The answer lies in two related phenomena: overparameterization and the lottery ticket hypothesis. Both provide complementary explanations for the same empirical observation: trained networks contain enormous amounts of redundancy.

Neural networks are routinely trained with far more parameters than are theoretically necessary to represent the solution. A binary text classifier on a 100-dimensional feature space does not need millions of parameters to learn a decision boundary. A language model trained on the English Wikipedia does not need 175 billion parameters to represent the statistical regularities of English text. Yet we train these oversized networks, and they work extraordinarily well.

This overparameterization is not accidental; it serves a essential purpose during training. Gradient descent on a highly overparameterized network is much easier to optimize than on a minimal network. The loss field of an overparameterized model has fewer sharp local minima and saddle points, and random initialization is far more likely to land in a region from which gradient descent can find a good solution. The gradient signal is also more reliable: each parameter update has a small relative impact on the overall loss, making optimization stable and predictable. In compact, under-parameterized networks, each weight carries more responsibility, the loss field is sharper, and optimization is more sensitive to initialization and learning rate.

Once training is complete, however, the extra capacity is no longer needed. The network has converged to a solution that uses only a fraction of its theoretical expressiveness. The surplus parameters are effectively frozen echoes of the training dynamics, not active contributors to the learned function. They can be removed without meaningful degradation because the surviving parameters already encode everything the model needs.

Empirical evidence strongly supports this view. Researchers have pruned 90 to 99 percent of parameters from trained image classifiers and language models while retaining nearly all predictive accuracy. The BERT model, with 110 million parameters, can be pruned to 40 percent sparsity with less than one point of accuracy loss on GLUE benchmarks. GPT-2 shows similar resilience. The trained network is not merely large; it is the compressed solution wrapped in an enormous amount of structural redundancy.

The redundancy shows up in several forms. Many weights in a trained network are near-zero, contributing negligible output to any downstream computation. Other weights are highly correlated with nearby weights in the same layer, meaning multiple weights encode nearly identical information. Some neurons in fully-connected layers fire rarely, activating only on a small fraction of inputs, and removing them affects only those rare cases. All of these patterns are exploitable by pruning.

A more formal account of why pruning works comes from the Lottery Ticket Hypothesis, proposed by Frankle and Carlin in 2019. The hypothesis states: a randomly initialized dense network contains a subnetwork (a "winning ticket") that, when trained in isolation from the same initial weights, can match the accuracy of the full network in comparable training time.

The key phrase is "from the same initial weights." The hypothesis is not simply that a small subnetwork exists within the full network after training. It is that there exists a specific subnetwork whose initial weights, drawn from the same random initialization as the full network, are a good starting point for learning. The initial random values of the winning ticket's weights matter. If you take the winning ticket's architecture but re-initialize its weights randomly, it may not train as well.

The lottery ticket metaphor is apt. When you buy a lottery ticket, you do not know at purchase time which ticket will win. Similarly, when you randomly initialize a large network, the winning ticket is hidden inside it, but you cannot identify it in advance. You have to run the full training process, then look backward to find which subnetwork was doing the real work.

The practical implications are significant. First, it explains why pruning works: you are essentially discovering the winning ticket after the fact. Second, it motivates a pruning methodology: train the full network, prune to find the winning ticket structure, then potentially retrain that structure from the original initialization (a strategy called weight rewinding, which we cover later in the chapter). Third, it suggests that network architecture is not the only thing that matters; the specific pattern of which weights are nonzero is itself a form of architecture.

The hypothesis also has notable limitations. Frankle and Carlin's original experiments worked well on small networks. Scaling the approach to large transformer models revealed complications: the winning ticket at initialization is harder to find for very deep networks, and the original strong form of the hypothesis (rewinding to step 0) sometimes fails, though a weaker form (rewinding to an early-training checkpoint rather than initialization) tends to work. We revisit this in the pruning schedules section.

Transformer-based language models exhibit a particularly interesting form of redundancy. Attention heads within the same layer often compute related or partially redundant functions. In multi-head self-attention, each head attends to different positions, but heads within a layer frequently specialize in similar syntactic or positional patterns, and removing several heads from the same layer often has minimal impact on performance. Michel et al. (2019) demonstrated that BERT retains competitive performance after removing 50 to 60 percent of attention heads from most layers. This suggests that language-model redundancy appears throughout the architecture and is especially concentrated in the attention mechanism.

Feed-forward layers in transformers also contain substantial redundancy. The intermediate dimension of a transformer's feed-forward sublayer is typically four times the model dimension, a ratio that comes from the original transformer paper without a strong theoretical justification. Subsequent work has shown that this 4x ratio is conservative; models trained with smaller feed-forward dimensions or with a fraction of the feed-forward neurons removed retain most of their capability.

Understanding these structural patterns is what distinguishes naive magnitude pruning from principled structured pruning for transformers. The fundamentals covered in this chapter apply uniformly; the next chapter specializes them to the transformer architecture.

A pruning criterion is a function that assigns an importance score to each element being considered for removal. Elements with low importance scores are pruned first. The choice of criterion has a large effect on how gracefully a model degrades as sparsity increases, and choosing the right criterion for your use case is as important as choosing the right schedule.

Several pruning criteria are commonly used in practice, ranging from the extremely simple to the computationally intensive.

The simplest and most widely used criterion is weight magnitude. The assumption is direct: weights close to zero contribute little to the network's output and can be safely removed.

Formally, for a weight $w_{ij}$ connecting neuron $i$ in layer $l$ to neuron $j$ in layer $l+1$, the importance score is:

where:

• $w_{ij}$: the scalar weight value connecting neurons $i$ and $j$
• $s_{ij}$: the importance score assigned to that weight, where higher scores indicate greater importance

Weights are sorted by importance score in ascending order. The bottom $k\%$ by score, where $k$ is the target sparsity ratio, are pruned by setting them to zero.

The simplicity of magnitude pruning belies its effectiveness. Despite requiring no gradient computation and no forward pass over a dataset, magnitude pruning competes well with more sophisticated methods on most standard benchmarks. Han et al.'s original 2015 results pruning LeNet and AlexNet used magnitude pruning almost exclusively. BERT pruning experiments consistently find that magnitude-based methods achieve accuracy within one to two points of gradient-based methods at equivalent sparsity levels, at a fraction of the computational cost.

Why does such a simple heuristic work so well? The answer is that gradient descent implicitly drives unimportant weights toward zero. When a weight is irrelevant to the loss, the gradient pushes it in directions that do not systematically increase its magnitude, and weight decay (L2 regularization) actively shrinks it. By the end of training, weights that the network does not use have been nudged toward zero. Magnitude pruning is, in a sense, reading the signal that the optimization process has already embedded in the weight values.

Magnitude pruning has a meaningful weakness: it ignores context. A small weight might contribute significantly if it lies on an information-critical path through the network. A neuron that connects two important computation chains might have small incoming weights because the downstream layer has large weights that amplify its output. Removing the small upstream weight then indirectly damages the large downstream computation. Magnitude pruning cannot detect this kind of dependency. Gradient-based and second-order methods are designed to address exactly this limitation.

Another subtlety is the choice between local and global magnitude pruning. Local pruning applies a sparsity threshold within each layer independently, making sure each layer reaches the same sparsity ratio. Global pruning ranks all weights in the network together, regardless of layer, and prunes the bottom fraction globally. In practice, global pruning typically produces better results because some layers are more critical than others. The softmax layer and the embedding layer may be very sensitive to pruning, and global pruning naturally prunes them less aggressively than layers with more redundancy.

A more detailed criterion uses gradient information alongside weight magnitude. The intuition is that the gradient of the loss with respect to a weight tells us how much the loss would change if that weight changed slightly. A weight with a large gradient magnitude is currently being updated aggressively and is likely doing important work; a weight with a near-zero gradient is not contributing to loss reduction and may be dormant.

One concrete formulation is the gradient-magnitude product, sometimes called the saliency score:

where:

• $w_{ij}$: the current weight value
• $\frac{\partial \mathcal{L}}{\partial w_{ij}}$: the partial derivative of the loss $\mathcal{L}$ with respect to $w_{ij}$
• $s_{ij}$: the importance score, capturing both the current size and the sensitivity of the weight

This product has a natural interpretation via a first-order Taylor expansion of the loss. If we were to set weight $w_{ij}$ to zero (a change of $-w_{ij}$ from its current value), the change in loss would be approximately:

The absolute value of this quantity, $|w_{ij} \cdot \partial \mathcal{L}/\partial w_{ij}|$, estimates how much zeroing the weight would change the loss. A small product means removing the weight has little impact on the loss. This is a stronger criterion than magnitude alone because it accounts for gradient context: a small weight that sits at a high-gradient position is treated as more important than a small weight at a flat region.

The gradient-magnitude product requires one forward pass and one backward pass over a calibration dataset to compute the gradients. This is more expensive than pure magnitude pruning but still tractable for most models. The gradient values change during fine-tuning, so this criterion is typically computed once on a representative batch before each pruning step.

One subtlety is that gradient values can be noisy, especially in mini-batch training. Using gradients averaged over multiple batches or an entire epoch reduces noise and gives more reliable importance estimates. Some implementations accumulate the gradient signal over a window of training steps rather than using a single batch.

A theoretically stronger criterion uses second-order information: the curvature of the loss field. The main insight is that the first-order Taylor approximation ignores the curvature of the loss, which can be significant. Two weights with identical gradient-magnitude products might have very different actual impacts on the loss if the loss field curves steeply at one location and gently at the other.

The Optimal Brain Damage (OBD) method, published by LeCun, Denker, and Solla in 1990, introduced the use of the Hessian for weight importance. OBD derives saliency scores by computing the second-order change in loss when each weight is set to zero. Under the assumption that the Hessian is diagonal (each weight's effect is independent), the OBD saliency score for weight $w_{ij}$ is:

where:

• $\frac{\partial \mathcal{L}}{\partial w_{ij}}$: the gradient of the loss with respect to $w_{ij}$
• $H_{ij,ij}$: the diagonal element of the Hessian matrix corresponding to $w_{ij}$, the second derivative of the loss with respect to $w_{ij}$ (also called the curvature)
• $s_{ij}$: the estimated increase in loss from zeroing weight $w_{ij}$

The denominator $H_{ij,ij}$ acts as a normalizer. A weight with a large gradient but flat curvature is riskier to prune than a weight with the same gradient but steep curvature, because in the steep case, nearby weights can more easily compensate via fine-tuning. The OBD formula captures this intuition.

The successor method, Optimal Brain Surgeon (OBS), removes the diagonal approximation and accounts for off-diagonal Hessian terms. OBS also derives the optimal weight adjustment for the remaining weights after each pruning step, so accuracy loss is minimized at each removal. OBS is more accurate than OBD but requires solving a linear system involving the full Hessian inverse, which is intractable for networks with millions of parameters. Various approximations have been developed to make OBS tractable at scale.

Modern methods like SparseGPT (Frantar and Alistarh, 2023) apply OBS-style ideas to prune large language models in a single pass without any fine-tuning, using efficient block-wise Hessian approximations that scale to billions of parameters. SparseGPT can prune GPT-3 variants to 50% sparsity with less than one percent increase in perplexity. This demonstrates that second-order methods, when carefully implemented, substantially outperform magnitude-based methods at high sparsity for large models.

The computational cost of second-order methods is their main barrier. Computing and storing the Hessian for a modern language model is infeasible without approximations. Diagonal Hessian estimates (from Fisher information or empirical risk curvature) are tractable but imprecise. Block-wise approaches that compute the Hessian for small groups of layers independently are a practical middle ground. As hardware and software improve, expect second-order pruning methods to become more common in production pipelines.

For neurons or attention heads rather than individual weights, a natural criterion is the magnitude of the neuron's activation across a representative dataset. A neuron that consistently produces near-zero activations regardless of the input is not contributing meaningful computation and can be safely removed.

Given a calibration dataset $\mathcal{D}$ of $N$ examples, the importance score for neuron $j$ is:

where:

• $a_j(x)$: the activation of neuron $j$ on input $x$
• $N$: the number of examples in the calibration dataset
• $s_j$: the mean absolute activation, averaged over the dataset

This criterion does not require gradients and is cheap to compute: a single forward pass over the calibration data gives activations for all neurons. The calibration dataset should be representative of the model's deployment distribution; using an unrepresentative set can lead to removing neurons that are important for common inputs but happen to fire rarely on the calibration examples.

Activation-based criteria are particularly useful when pruning at a coarser granularity than individual weights, such as pruning entire attention heads or feed-forward neurons in a transformer. For attention head pruning, the activation criterion can be adapted to measure the average entropy of the head's attention distribution or the variance of its outputs across inputs, both of which capture how actively the head is being used.

The four criteria we have covered represent a spectrum from simple to sophisticated. In practice, the choice depends on the available compute budget and the target sparsity level.

At low to moderate sparsity (below 50%), magnitude pruning often achieves near-optimal results at minimal cost. The gradient information encoded in the weight values is sufficient to identify clearly unimportant parameters. At high sparsity (above 70%), gradient-based and second-order methods show more significant advantages because the decision of which weights to keep becomes more consequential. At 90% sparsity, removing the wrong 9% can cause much larger accuracy drops than at 50% sparsity.

For structured pruning of transformer components (heads, neurons, layers), activation-based criteria are often preferred because they directly measure whether a structural unit is doing useful computation, rather than measuring individual weight importance within the unit.

A fundamental design choice in any pruning system is the granularity of what gets removed. This is the distinction between structured and unstructured pruning, and it has major practical consequences for both accuracy and inference speed.

Unstructured pruning removes individual weights anywhere in the network, regardless of their position. The result is a sparse weight matrix: a matrix with the same shape as the original but containing many zeros scattered throughout in an irregular pattern.

Unstructured pruning is maximally flexible. Because it can remove any individual weight, it can achieve high sparsity ratios with minimal accuracy loss. The pruning criterion operates independently on each weight, and the pattern of zeros is irregular and input-specific. A weight that is unimportant for the given task gets removed; one that is important stays, regardless of where it sits in the matrix.

The critical limitation of unstructured pruning is that it does not translate directly into computational speedups on standard hardware. Modern CPUs and GPUs are designed around dense matrix multiplication. They process entire matrices at once using highly optimized BLAS (Basic Linear Algebra Subprograms) routines that assume contiguous, regular data. A matrix that is 90% zeros but stored in the same dense format as a fully dense matrix requires the same number of memory reads and the same number of floating-point operations to multiply. The zeros do not make multiplication faster unless special sparse matrix formats and hardware support are used to skip the zero multiplications.

Achieving real speedups from unstructured pruning requires sparse matrix representations, such as CSR (compressed sparse row format) or COO (coordinate list format), combined with hardware or software capable of skipping zero multiplications. This is increasingly available on specialized AI accelerators, but less common on the general-purpose hardware deployed at scale in production inference infrastructure. The result is a frustrating disconnect: an unstructured pruning approach can reduce model file size and memory footprint by 10x while producing almost no improvement in inference latency.

When does unstructured pruning make sense? First, when the deployment target explicitly supports sparse computation. NVIDIA's Sparse Tensor Cores, newer NPUs, and some specialized inference chips can take advantage of sparsity even in irregular patterns. Second, when the goal is to reduce memory bandwidth rather than compute throughput. Moving a 90% sparse model over a network or loading it from disk is faster even if inference is not, which matters for applications with large model download sizes. Third, when unstructured pruning is combined with weight encoding techniques that efficiently represent sparse matrices on disk.

Structured pruning removes entire groups of weights: whole neurons, attention heads, convolutional filters, or entire transformer layers. The resulting network has the same dense structure as the original, but with fewer neurons, heads, or layers. Because the remaining weights still form regular, dense matrices, standard hardware can execute the computation efficiently without any special sparse support.

The tradeoff is accuracy. Structured pruning is coarser than unstructured pruning. Removing an entire neuron removes all the weights connected to it, some of which might individually be important. The information that neuron encoded is lost entirely, and this can hurt accuracy more than removing the same number of weights spread diffusely across many neurons via unstructured pruning. Structured pruning compresses less gracefully: for the same target sparsity, it typically causes larger accuracy drops than unstructured pruning.

The choice between structured and unstructured pruning often comes down to the deployment target. For mobile CPUs and general-purpose hardware, structured pruning is preferable because it produces a physically smaller, faster model with no infrastructure changes required. A model with 30% fewer attention heads and 40% fewer feed-forward neurons just runs faster on any hardware, no sparse compute support needed. For specialized inference hardware that supports sparse computation, unstructured pruning is preferable because it achieves higher compression ratios at the same accuracy level.

A practical heuristic: if you need to deploy on standard hardware today, use structured pruning. If you are building an infrastructure stack around specialized sparse hardware, invest in unstructured pruning pipelines.

A middle ground between structured and unstructured pruning is semi-structured pruning, also called N:M sparsity. In N:M sparsity, exactly $N$ out of every $M$ consecutive weights are pruned. For example, 2:4 sparsity means exactly 2 out of every 4 consecutive weights are zero, achieving exactly 50% sparsity in a pattern that modern NVIDIA Ampere GPUs can accelerate natively.

NVIDIA's Sparse Tensor Cores on Ampere and later architectures are designed specifically for 2:4 sparsity. They can execute the sparse matrix multiplication at twice the throughput of the equivalent dense computation, while storing only the nonzero values and a compact index structure that records which positions within each 4-element group are nonzero (requiring only 2 bits per group of 4). This gives N:M sparsity the best of both worlds: measurable hardware speedups without the coarseness of fully structured pruning.

The constraint that sparsity must be exactly N:M (not more, not less, and in consecutive groups) is limiting. Unlike unstructured pruning, you cannot freely choose which weights to zero: you must remove exactly $N$ from each group of $M$. This forces some important weights to be removed (if they happen to be the $N$ smallest in their group of $M$) while preserving some unimportant weights (if they are among the $M-N$ largest in their group). The constraint introduces a mild accuracy penalty relative to unstructured pruning at the same sparsity ratio, but the hardware speedup makes it worthwhile for NVIDIA GPU deployments.

Selecting which $N$ weights to remove from each group is itself a small optimization problem. Magnitude-based selection within each group is the default, but gradient-based selection within groups can achieve slightly better accuracy when the within-group importance scores are more heterogeneous.

How and when you remove weights matters as much as which weights you remove. The pruning schedule controls the trajectory from a dense, fully trained model to a sparse, pruned model. Different schedules make different tradeoffs between compute cost, final accuracy, and implementation complexity.

The simplest schedule is one-shot pruning: train the model to convergence, prune to the target sparsity in a single step, then fine-tune on the training data to recover accuracy. This is the most conceptually straightforward approach and is a useful baseline for evaluating other schedules.

One-shot pruning works reasonably well at moderate sparsity levels, up to around 50 to 60 percent, but becomes increasingly problematic at high sparsity. When a large fraction of weights are removed simultaneously, the model's ability to recover through fine-tuning is limited. The network can no longer represent many of the functions it previously encoded, and fine-tuning on the remaining sparse network may not be sufficient to compensate.

Think of it like removing pieces from a jigsaw puzzle and asking someone to complete it with the remaining pieces. If you remove 20% of the pieces, completion may still be feasible; the remaining pieces constrain the solution well. If you remove 80% of the pieces, the remaining pieces are insufficient to reconstruct the original image, and the best you can do with fine-tuning is to fill in the gaps as creatively as possible.

One-shot pruning shines when computational cost is the primary constraint. A single fine-tuning run is inexpensive relative to the full original training. If moderate sparsity is sufficient for the deployment target, one-shot pruning achieves a good accuracy-efficiency tradeoff with minimal additional compute.

Iterative pruning alternates between pruning and fine-tuning across multiple cycles. A typical schedule looks like:

1. Train the model to convergence on the full dataset.
2. Prune a fraction of the target sparsity (for example, prune 10% of weights to reach 10% sparsity).
3. Fine-tune the pruned model for a few thousand steps.
4. Repeat steps 2 and 3 until the target sparsity is reached.

Iterative pruning consistently outperforms one-shot pruning at the same final sparsity level. The reason is that each fine-tuning phase allows the remaining weights to adjust and compensate for what was removed before the next round of pruning. The model adapts incrementally rather than absorbing a single catastrophic reduction. At each step, the model is at a working, recoverable state before the next pruning is applied.

The accuracy advantage of iterative over one-shot pruning grows with the target sparsity. At 50% sparsity, iterative pruning may achieve one or two additional percentage points of accuracy. At 90% sparsity, the gap can be much larger, with one-shot pruning sometimes failing to recover meaningful accuracy while iterative pruning maintains reasonable performance.

The main disadvantage of iterative pruning is cost. Each prune-then-finetune cycle requires additional compute. For very large models, even a single fine-tuning run can take days on a cluster of GPUs, and iterative pruning might require five to twenty such cycles. The compute budget for pruning a large language model via iterative pruning is substantial, sometimes comparable to the original pretraining cost of a smaller model. This is one reason why the field has invested heavily in one-shot second-order methods like SparseGPT: they achieve high-quality pruning without the repeated fine-tuning cycles.

Gradual pruning (also called polynomial decay pruning) provides a continuous schedule that increases sparsity smoothly from zero to the target over the course of a single training run. Rather than alternating discrete prune-and-finetune phases, gradual pruning applies small incremental pruning steps at regular intervals during training, so pruning and learning happen simultaneously.

A common schedule increases the target sparsity $s(t)$ at training step $t$ according to:

where:

• $s_f$: the final target sparsity (for example, 0.9 for 90% sparsity)
• $t_0$: the training step at which pruning begins (often after an initial warmup period)
• $n$: the total number of pruning steps planned
• $\Delta t$: the interval between consecutive pruning steps
• $t$: the current training step

The cubic polynomial factor $\left(1 - (1 - \cdot)^3\right)$ means sparsity increases rapidly at first and then slows as it approaches the target. This front-loading of pruning gives the model more fine-tuning time at high sparsity levels, where recovery is most needed. The logic is that the first pruning steps remove the clearest redundancies, which the model can compensate for quickly, while the later pruning steps at high sparsity are more damaging and require more recovery time.

Gradual pruning integrates cleanly with standard training loops. You do not need separate training and fine-tuning phases; instead, you insert a pruning step every $\Delta t$ gradient updates. The model never experiences a large discontinuous change in its parameter values, which keeps optimization stable. It is the default in many production pruning pipelines, particularly those based on the TensorFlow Model Optimization Toolkit and the Hugging Face optimum library.

One important detail of gradual pruning is the choice of $t_0$, the step at which pruning begins. Starting pruning from the very first step is generally suboptimal, because the model has not yet learned a useful representation, and the importance scores computed at initialization are not meaningful. Starting pruning after a brief warmup period, typically 10 to 20% of total training, gives the model time to develop stable weight magnitudes before the pruning criterion is applied.

Motivated by the Lottery Ticket Hypothesis, a more elaborate schedule involves weight rewinding. After pruning the trained model to identify which weights to keep, instead of continuing from the current weights, the surviving weights are reset to their values from an early checkpoint (for example, after only 1,000 to 10,000 training steps). Only the pruning mask, indicating which weights are zero, is retained from the trained model. The model is then retrained from this early checkpoint with the pruning mask in place.

The intuition is that the winning ticket subnetwork trains better when started from its original early-training weights rather than the fully trained weights accumulated over the entire training run. Full training can cause surviving weights to partially compensate for the weights that will eventually be pruned, encoding information in ways that are hard to disentangle. By rewinding to an early checkpoint and retraining, the subnetwork gets a cleaner start: it develops its representations from scratch, knowing from the beginning which connections are available to it.

In practice, rewinding to step 0 (true initialization rewinding) works best on small networks and fails on large ones. Rewinding to a small positive step, such as after 1,000 steps (around 1 to 5% of total training), works reliably on deep networks including transformers. The key insight is that a few training steps are enough to find a starting point that is significantly better than pure initialization, while still being "early enough" that the subnetwork can develop useful representations from that starting point.

Weight rewinding is more expensive than standard one-shot pruning: after identifying the pruning mask, you must retrain from the early checkpoint, which is nearly as expensive as the original training run. The payoff is that rewound lottery ticket subnetworks sometimes generalize better than subnetworks fine-tuned from the pruned endpoint, particularly when the fine-tuning data distribution differs from the pretraining distribution.

To make these concepts concrete, consider a small fully connected layer with a $3 \times 4$ weight matrix:

This matrix has 12 weights total, connecting 4 input neurons to 3 output neurons. We want to apply 50% unstructured magnitude pruning, meaning we will zero out the 6 weights with the smallest absolute values.

The absolute values in ascending order are: 0.01, 0.01, 0.02, 0.03, 0.05, 0.05, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9. The threshold at 50% sparsity falls at $|w| = 0.05$ (the 6th smallest value). All weights with $|w_{ij}| \leq 0.05$ are zeroed. The surviving weights form the pruned matrix:

Six out of twelve weights are now exactly zero. The matrix has the same shape as before but contains 50% zeros. In a sparse storage format like CSR, we store only 6 nonzero values plus their column indices, roughly halving the storage requirement. On hardware that supports sparse computation, the matrix-vector product $W_{\text{pruned}} \mathbf{x}$ can skip the 6 zero multiplications.

Notice that the pruning preserved the structure of the large weights. The values 0.8, 0.7, 0.9, 0.5, 0.6, and 0.4 all survived, while the cluster of near-zero values (0.01, 0.02, 0.03, etc.) were removed. The large weights are likely encoding the most important signal, and the small weights are noise.

For structured pruning on the same matrix, we would instead consider removing entire rows, where each row corresponds to one output neuron. The importance score for each row is its L2 norm:

• Row 1: $\sqrt{0.8^2 + 0.02^2 + 0.5^2 + 0.01^2} \approx \sqrt{0.64 + 0.0004 + 0.25 + 0.0001} \approx 0.94$
• Row 2: $\sqrt{0.3^2 + 0.7^2 + 0.01^2 + 0.6^2} \approx \sqrt{0.09 + 0.49 + 0.0001 + 0.36} \approx 0.96$
• Row 3: $\sqrt{0.05^2 + 0.4^2 + 0.9^2 + 0.03^2} \approx \sqrt{0.0025 + 0.16 + 0.81 + 0.0009} \approx 0.99$

To prune 33% of neurons (removing 1 of 3 rows), we remove Row 1, which has the smallest norm. The result is a $2 \times 4$ weight matrix, not a $3 \times 4$ matrix with a zero row. This reduces the network's hidden dimension. Any matrix multiplications downstream of this layer also shrink, because the output dimension of this layer drops from 3 to 2. The result is a model that runs faster on any hardware.

The contrast between the two pruned matrices illustrates the core tradeoff. The unstructured-pruned $3 \times 4$ matrix and the structured-pruned $2 \times 4$ matrix both represent roughly the same amount of removed information, but only the structured-pruned matrix produces a smaller computation graph.

Let us implement the key components of a pruning pipeline using PyTorch's built-in pruning utilities. We will apply magnitude pruning to a small feedforward network and examine the sparsity-accuracy tradeoff as we increase the pruning ratio.

We start by importing the libraries and defining a small feedforward network for text classification.

The model has three fully connected layers. The first layer maps 100-dimensional TF-IDF-style features to 256 hidden units, the second halves that to 128, and the third produces the two-class logits. With three layers of sizes 100x256, 256x128, and 128x2, the weight matrices contain 25,600, 32,768, and 256 parameters respectively. The first two layers together account for over 98% of the model's parameters and are the primary targets for pruning.

We generate synthetic TF-IDF-like features and train the model to convergence before pruning. The synthetic task is simple: predict which of two word groups has a higher aggregate frequency in the input. This gives us a well-defined signal-to-noise ratio that makes the effect of pruning easy to interpret.

PyTorch's torch.nn.utils.prune module provides a clean API for applying pruning masks. We apply global unstructured magnitude pruning, which considers all weights across all specified layers together and prunes the globally smallest weights regardless of which layer they are in.

The global_unstructured function assigns a pruning mask to each parameter: a binary tensor of the same shape as the weight matrix, where 1 means "keep" and 0 means "prune." The actual weight values are not immediately zeroed; instead, PyTorch registers a forward hook that applies the mask on each forward pass. The prune.remove call makes the pruning permanent by baking the zeros directly into the weight tensor and removing the hook.

Fine-tuning the pruned model allows it to redistribute the workload among the remaining weights and recover lost accuracy. The surviving weights adjust their magnitudes to compensate for the information that was lost with the pruned weights. This is possible because neural networks are highly overdetermined: multiple weight configurations can encode the same function, and fine-tuning finds a new configuration within the constraints imposed by the sparse mask.

Now we implement a simplified gradual pruning schedule: instead of pruning everything at once, we prune incrementally during training. The model is re-initialized with random weights for this experiment so we can observe the full gradual pruning trajectory from scratch.

The key parameters for weight pruning are:

• amount (sparsity): The fraction of weights to prune (for example, 0.7 for 70% sparsity). Higher values compress more aggressively but risk greater accuracy loss. The optimal value depends on the model architecture, task, and available fine-tuning compute.
• pruning_method: The criterion for selecting which weights to remove. L1Unstructured applies magnitude pruning; RandomUnstructured provides a random baseline; custom methods can implement gradient-based or activation-based criteria.
• global_unstructured vs. per-layer: Global pruning considers all layers together and allocates sparsity based on magnitude across the entire network, naturally pruning sensitive layers less. Per-layer pruning applies a uniform sparsity ratio to each layer independently, which may under-prune critical layers and over-prune redundant ones.
• Fine-tuning learning rate: After pruning, fine-tuning with a reduced learning rate (typically 10 to 50% of the original) prevents large weight updates that could disrupt the remaining sparse structure.
• Gradual pruning interval: How often sparsity increases during gradual pruning. More frequent updates create a smoother sparsity trajectory; less frequent updates allow more recovery time between pruning steps.

One of the most important practical lessons when deploying pruned models is the gap between theoretical and realized efficiency. We have seen the accuracy-sparsity tradeoff; let us now examine the sparsity-efficiency gap more concretely.

The theoretical story is appealing: a model with 90% sparsity uses 10% of the non-zero parameters, so it should be roughly 10x smaller and faster. In practice, several obstacles prevent realizing this speedup on standard hardware.

Dense matrix multiply overhead. Standard deep learning frameworks represent weight matrices as dense tensors. A 90% sparse dense tensor occupies the same memory as a 100% dense tensor. When you call model(x) in PyTorch, it calls optimized BLAS routines that multiply every element, including the zeros. The zeros do not get skipped; they produce zero contributions that are added to the accumulator, consuming compute time and memory bandwidth.

Sparse format overhead. Converting to a sparse format (CSR, COO, or block-sparse) introduces representation overhead and complicates the matrix multiply. The sparse multiply must now read the column indices alongside the nonzero values, and the irregular memory access pattern defeats CPU and GPU cache prefetching optimizations. For moderate sparsity (below 90%), sparse matrix multiply can be slower than dense multiply because the overhead of managing the sparse format exceeds the savings from skipping zeros.

Hardware SIMD requirements. Modern CPUs execute matrix multiplications using SIMD (single instruction, multiple data) instructions that process 4, 8, or 16 values in parallel. These instructions require regular, predictable memory layouts. Unstructured sparsity breaks the regular layout, preventing effective SIMD utilization. Structured sparsity preserves the regular layout because entire neurons or rows are removed, not scattered individual elements.

The practical consequence is that unstructured pruning primarily reduces model file size and initial load time, not inference latency, on standard CPU and GPU hardware. For latency-sensitive applications, structured pruning or N:M sparsity on NVIDIA hardware is necessary to achieve real speedups.

The output confirms the sparsity-efficiency gap: despite 70% of weights being zero, inference time on this dense-format implementation is similar to the baseline. The zeros are being multiplied and added just like any other value. Achieving real speedups requires either converting to a sparse format, using structured pruning so the matrix itself is smaller, or deploying on hardware with native sparse compute support.

Pruning is a powerful technique, but it comes with real limitations that practitioners must manage carefully. These limitations help determine when pruning is the right tool and how to deploy it successfully.

The most significant practical limitation is the gap between sparsity and speedup. In theory, a model that is 90% sparse should be much faster. In practice, this speedup is rarely realized on standard hardware unless the sparsity is structured or the hardware explicitly supports sparse computation. Dense matrix multiplication is a solved problem with decades of optimized implementations: CPU BLAS and GPU cuBLAS are extraordinarily efficient at dense multiplies. Sparse matrix multiplication, especially with irregular sparsity patterns, is harder to optimize and achieves less consistent speedups. For many production deployments on CPUs and GPUs without sparse tensor core support, unstructured pruning reduces storage requirements and memory footprint but not inference latency. This is the central frustration of unstructured pruning: the model is smaller on disk, but not faster in production.

Fine-tuning after pruning requires access to the original training data or a representative proxy dataset. This creates a challenge for models trained on proprietary or privacy-sensitive data, where retraining with the original corpus may not be possible. Federated learning and differential privacy add further complications: if the training data is distributed across clients or cannot be centralized, standard fine-tuning-based pruning pipelines break down. Some researchers have explored data-free pruning methods that use synthetic or generated data for calibration, but these are less mature and typically achieve lower quality results than methods with access to real training data.

The optimal sparsity level depends heavily on the model architecture, the task, and the hardware target. There is no universal answer; every deployment scenario requires its own empirical sweep across sparsity levels to find the accuracy-efficiency frontier. This evaluation cost is non-trivial for large models where a single evaluation run takes hours and fine-tuning takes days. Researchers are developing principled methods for predicting the accuracy-sparsity tradeoff without running the full sweep, but this problem is largely unsolved.

Pruning also interacts in complex ways with other compression techniques. Quantization after pruning can introduce quantization errors into weights that the pruning phase assumed were important. The combination of pruning and low-bit quantization can produce accuracy drops that are larger than the sum of the drops from each technique applied independently, particularly at high sparsity and low bit-width. Some practitioners have found that distilling first to get a smaller, cleaner student model, then applying quantization and pruning to the student, gives better results than applying all three to the original model simultaneously.

Another subtle limitation concerns the stability of pruning masks. In iterative and gradual pruning, the set of weights that are zeroed can change between pruning steps as the surviving weights are updated by fine-tuning. A weight that was small enough to be pruned in step 1 might grow large enough to be important by step 3, but it was already zeroed and its gradient is no longer updated. This means that iterative pruning can accidentally discard weights that would have become important with continued training. Some approaches address this by allowing "regrowth": periodically unfreezing some zeroed weights and allowing them to recover if gradient information suggests they have become important. Sparse learning methods that explicitly combine pruning with weight regrowth (such as dynamic sparse training) are an active research direction.

Finally, pruning is most effective when the original model is significantly overparameterized relative to the task. A model trained at near-minimal capacity may have no redundancy to prune, and aggressive sparsity will immediately degrade performance. There is a practical sweet spot, typically the class of "large but not frontier" models, where pruning provides the best return on investment.

Pruning removes weights, neurons, or structural components from trained neural networks by exploiting the redundancy introduced during overparameterized training. The history of the technique spans from LeCun's Optimal Brain Damage in 1990 through the lottery ticket hypothesis and modern one-shot methods for billion-parameter language models.

The key concepts covered in this chapter are:

• Why pruning works: Trained networks are heavily overparameterized. The lottery ticket hypothesis provides a formal account: every large network contains a smaller subnetwork that can match the full network's accuracy when trained from the same initialization. Pruning finds this subnetwork after the fact.
• Pruning criteria determine which weights are removed. Magnitude pruning (removing the smallest weights by absolute value) is the simplest and most widely used. Gradient-based methods use the saliency score $|w \cdot \partial \mathcal{L} / \partial w|$ for more principled removal. Second-order methods using Hessian information (OBD, OBS, SparseGPT) make more accurate predictions about which weights are safe to remove, especially at high sparsity.
• Unstructured pruning removes individual weights at arbitrary positions, achieving high compression with minimal accuracy loss but requiring sparse compute support for real hardware speedups.
• Structured pruning removes entire neurons, heads, or layers, producing a smaller dense network that runs efficiently on standard hardware at the cost of higher accuracy loss per unit of compression.
• Semi-structured (N:M) pruning provides a middle ground: regular sparsity patterns that NVIDIA Ampere hardware can accelerate natively, giving measurable speedups while allowing finer-grained removal than fully structured pruning.
• Pruning schedules control the trajectory from dense to sparse. One-shot pruning applies the full reduction at once; iterative pruning alternates prune-and-finetune cycles; gradual pruning increases sparsity continuously during training. Iterative and gradual schedules consistently outperform one-shot pruning at high sparsity. Weight rewinding, motivated by the lottery ticket hypothesis, resets surviving weights to early-training checkpoints for potentially better subnetwork training.
• Fine-tuning after pruning is essential for recovering accuracy. The pruned model adjusts the magnitudes of surviving weights to compensate for the removed connections.
• The sparsity-efficiency gap: Achieving real hardware speedups from sparsity requires either structured pruning, N:M sparsity on supporting hardware, or sparse compute infrastructure. Unstructured pruning on standard hardware primarily reduces model size, not inference latency.

The next chapter on Structured Pruning applies these fundamentals specifically to transformer architectures, examining how to prune attention heads, feed-forward neurons, and entire layers while preserving as much of the model's language understanding as possible.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about pruning basics.