Pruning Basics › Structured vs. Unstructured Pruning

A fundamental design choice in any pruning system is the granularity of what gets removed. This is the distinction between structured and unstructured pruning, and it has major practical consequences for both accuracy and inference speed.

Unstructured pruning removes individual weights anywhere in the network, regardless of their position. The result is a sparse weight matrix: a matrix with the same shape as the original but containing many zeros scattered throughout in an irregular pattern.

Unstructured pruning is maximally flexible. Because it can remove any individual weight, it can achieve high sparsity ratios with minimal accuracy loss. The pruning criterion operates independently on each weight, and the pattern of zeros is irregular and input-specific. A weight that is unimportant for the given task gets removed; one that is important stays, regardless of where it sits in the matrix.

The critical limitation of unstructured pruning is that it does not translate directly into computational speedups on standard hardware. Modern CPUs and GPUs are designed around dense matrix multiplication. They process entire matrices at once using highly optimized BLAS (Basic Linear Algebra Subprograms) routines that assume contiguous, regular data. A matrix that is 90% zeros but stored in the same dense format as a fully dense matrix requires the same number of memory reads and the same number of floating-point operations to multiply. The zeros do not make multiplication faster unless special sparse matrix formats and hardware support are used to skip the zero multiplications.

Achieving real speedups from unstructured pruning requires sparse matrix representations, such as CSR (compressed sparse row format) or COO (coordinate list format), combined with hardware or software capable of skipping zero multiplications. This is increasingly available on specialized AI accelerators, but less common on the general-purpose hardware deployed at scale in production inference infrastructure. The result is a frustrating disconnect: an unstructured pruning approach can reduce model file size and memory footprint by 10x while producing almost no improvement in inference latency.

When does unstructured pruning make sense? First, when the deployment target explicitly supports sparse computation. NVIDIA's Sparse Tensor Cores, newer NPUs, and some specialized inference chips can take advantage of sparsity even in irregular patterns. Second, when the goal is to reduce memory bandwidth rather than compute throughput. Moving a 90% sparse model over a network or loading it from disk is faster even if inference is not, which matters for applications with large model download sizes. Third, when unstructured pruning is combined with weight encoding techniques that efficiently represent sparse matrices on disk.

Structured pruning removes entire groups of weights: whole neurons, attention heads, convolutional filters, or entire transformer layers. The resulting network has the same dense structure as the original, but with fewer neurons, heads, or layers. Because the remaining weights still form regular, dense matrices, standard hardware can execute the computation efficiently without any special sparse support.

The tradeoff is accuracy. Structured pruning is coarser than unstructured pruning. Removing an entire neuron removes all the weights connected to it, some of which might individually be important. The information that neuron encoded is lost entirely, and this can hurt accuracy more than removing the same number of weights spread diffusely across many neurons via unstructured pruning. Structured pruning compresses less gracefully: for the same target sparsity, it typically causes larger accuracy drops than unstructured pruning.

The choice between structured and unstructured pruning often comes down to the deployment target. For mobile CPUs and general-purpose hardware, structured pruning is preferable because it produces a physically smaller, faster model with no infrastructure changes required. A model with 30% fewer attention heads and 40% fewer feed-forward neurons just runs faster on any hardware, no sparse compute support needed. For specialized inference hardware that supports sparse computation, unstructured pruning is preferable because it achieves higher compression ratios at the same accuracy level.

A practical heuristic: if you need to deploy on standard hardware today, use structured pruning. If you are building an infrastructure stack around specialized sparse hardware, invest in unstructured pruning pipelines.

A middle ground between structured and unstructured pruning is semi-structured pruning, also called N:M sparsity. In N:M sparsity, exactly $N$ out of every $M$ consecutive weights are pruned. For example, 2:4 sparsity means exactly 2 out of every 4 consecutive weights are zero, achieving exactly 50% sparsity in a pattern that modern NVIDIA Ampere GPUs can accelerate natively.

NVIDIA's Sparse Tensor Cores on Ampere and later architectures are designed specifically for 2:4 sparsity. They can execute the sparse matrix multiplication at twice the throughput of the equivalent dense computation, while storing only the nonzero values and a compact index structure that records which positions within each 4-element group are nonzero (requiring only 2 bits per group of 4). This gives N:M sparsity the best of both worlds: measurable hardware speedups without the coarseness of fully structured pruning.

The constraint that sparsity must be exactly N:M (not more, not less, and in consecutive groups) is limiting. Unlike unstructured pruning, you cannot freely choose which weights to zero: you must remove exactly $N$ from each group of $M$. This forces some important weights to be removed (if they happen to be the $N$ smallest in their group of $M$) while preserving some unimportant weights (if they are among the $M-N$ largest in their group). The constraint introduces a mild accuracy penalty relative to unstructured pruning at the same sparsity ratio, but the hardware speedup makes it worthwhile for NVIDIA GPU deployments.

Selecting which $N$ weights to remove from each group is itself a small optimization problem. Magnitude-based selection within each group is the default, but gradient-based selection within groups can achieve slightly better accuracy when the within-group importance scores are more heterogeneous.