Structured Pruning: Head, Layer, and Width Pruning

Modern language models have grown to sizes where deploying them in production is difficult. A model with billions of parameters demands significant memory, processing time, and energy on every inference call. Knowledge distillation, which we covered earlier in this part, trains a smaller model to mimic a larger one. Quantization reduces the numerical precision of weights. Pruning takes a different approach: it removes parts of an existing model entirely, creating a physically smaller architecture that performs nearly as well as the original.

Pruning broadly divides into two families. Unstructured pruning, covered in the previous chapter, sets individual weights to zero without changing the model's computational graph. The weight matrices remain the same shape, which means standard GPU kernels see no speedup unless you use specialized sparse computation hardware. Structured pruning, the focus of this chapter, removes entire structural components: whole attention heads, entire transformer layers, or full rows and columns from weight matrices. Because the model architecture changes shape, structured pruning produces real speedups on any hardware without special sparse kernels.

The distinction matters enormously in practice. When you remove a single attention head from a 12-head transformer layer, the resulting layer has 11 heads and its weight matrices shrink proportionally. The forward pass takes less memory and fewer FLOPs on every call. When you remove a full transformer layer from a 12-layer model, every inference call skips that layer entirely. These are concrete, measurable speedups that show up in production latency metrics. Structured pruning is not about theoretical sparsity; it is about real hardware-visible compression that you can benchmark directly.

The field of structured pruning for transformers emerged from broader work on network pruning in convolutional networks, where researchers found that removing entire filters rather than individual weights produced better hardware acceleration. When BERT arrived in 2018 and researchers began analyzing its attention patterns, it quickly became clear that the same principle applied: many heads were redundant, many layers were near-identity, and removing them in bulk produced models that were both smaller and faster without catastrophic quality loss. This observation motivated a wave of papers on head pruning and layer dropping that established the foundational techniques we will study here.

A recurring observation in transformer research is that not all components contribute equally to model quality. Many attention heads appear redundant: when researchers examined BERT's attention patterns, they found that many heads learned similar patterns, and individual heads could be masked off at test time with minimal quality loss. The 2019 paper "Are Sixteen Heads Really Better than One?" by Michel and colleagues showed that many BERT heads could be removed without any fine-tuning and the model would retain most of its performance. This result was surprising at the time because it suggested that the model was substantially over-provisioned for most tasks.

Similarly, analysis of layer-by-layer representations in deep transformers shows that middle layers often produce very similar hidden states, suggesting many computations could be skipped. Gromov and colleagues demonstrated in 2024 that in large language models like Llama 2, angular distances between consecutive layer representations vary dramatically across the depth of the network. Early layers perform large, varied transformations. Late layers also apply significant transformations, particularly those responsible for mapping representations to output token probabilities. But a set of middle layers often pass the representation through nearly unchanged.

This redundancy is not an accident. Large models are typically overparameterized relative to any specific downstream task. During pretraining on broad corpora, the model learns many different capabilities, some of which are irrelevant for a particular application. Fine-tuning on a narrow task does not eliminate the circuits dedicated to irrelevant capabilities; it just reduces their influence on the output. Structured pruning provides a systematic way to find and excise these underutilized components.

The key insight driving all structured pruning methods is that you need a criterion for measuring importance. An importance score tells you which heads, layers, or neurons matter most. Once you have scores, you can rank all components and remove the least important ones until you reach a target size or quality threshold. The challenge is computing reliable importance scores without running an exhaustive ablation study, which would require removing each component individually and evaluating quality, an approach that does not scale to models with hundreds of layers and thousands of heads.

There are three main families of importance criteria. Perturbation-based methods measure the actual quality change from removing a component. Gradient-based methods use the gradient of the loss with respect to component outputs to estimate sensitivity. Activation-based methods measure how active a component is on representative inputs. Each has different computational costs and different accuracy characteristics, and practitioners often combine them.

Transformer attention layers contain multiple parallel attention heads. Each head applies a separate set of queries, keys, and values to the input, then projects the results back to the model dimension. The motivation for multiple heads is that different heads can attend to different aspects of the input: one head might track syntactic subject-verb dependencies while another tracks positional relationships, and another might focus on coreference.

The multi-head attention computation concatenates the outputs of all heads and applies a final linear projection. The full attention output for a layer is:

where each head computes:

where:

• $H$ is the number of heads
• $d_k$ is the head dimension
• $\mathbf{W}_Q^{(h)}, \mathbf{W}_K^{(h)}, \mathbf{W}_V^{(h)} \in \mathbb{R}^{d_{\text{model}} \times d_k}$ are the query, key, and value projections for head $h$
• $\mathbf{W}_O \in \mathbb{R}^{(H \cdot d_k) \times d_{\text{model}}}$ is the output projection that assembles all heads

Because each head's contribution is additive after the output projection, you can set any head's output to zero and the remaining heads still produce a valid (though potentially degraded) result. This additive structure is precisely what makes head pruning possible: it is a well-defined operation with predictable effects.

The most direct importance score for a head is its sensitivity to removal. You can estimate this by computing how much the loss increases when you zero out that head's output. Formally, for head $h$ in layer $l$, define the importance score as:

where:

• $\mathcal{L}(\theta)$ is the loss with all heads present
• $\mathcal{L}(\theta \setminus h_l)$ is the loss with head $h$ in layer $l$ zeroed out
• $|\cdot|$ is the absolute value, giving a non-negative importance score

Computing this exactly requires a separate forward pass for each head, which is expensive. A first-order approximation uses gradient information instead. For a head parameterized by its output projection weights $\mathbf{W}_O^{(h,l)}$, the gradient-based importance score is:

where:

• $\frac{\partial \mathcal{L}}{\partial \mathbf{W}_O^{(h,l)}}$ is the gradient of the loss with respect to the head's output projection weights
• $\odot$ is element-wise multiplication
• $\|\cdot\|_F$ is the Frobenius norm, giving the overall magnitude of the score

This gradient-weight product, sometimes called the Taylor expansion criterion, approximates the change in loss from zeroing out the head. Intuitively, a head with large weights and large gradients is one where the model is heavily relying on that head and adjusting it actively. Large values indicate that the weights are large and their gradients are large, meaning the head contributes substantially to the loss. Small values indicate the head could be removed with little effect. This approximation is derived from a first-order Taylor expansion of the loss around the current parameters, treating the removal of the head as a perturbation.

A simpler but surprisingly effective approach uses attention head activation norms. For each head, compute the mean magnitude of its output across a calibration dataset:

where $\text{head}_h^{(l)}(x)$ is the output vector produced by head $h$ in layer $l$ for input $x$, before it is passed through the output projection.

Activation norm-based importance has the advantage that it requires no gradient computation and can be collected during normal inference. The disadvantage is that it measures how active a head is, not how important it is to the final output. A very active head whose output is largely cancelled by the output projection could score high by this criterion yet contribute little to the model's predictions. In practice, gradient-based scores tend to be more accurate, but activation norms are a useful fast approximation.

Once importance scores are computed, pruning proceeds by selecting the heads to remove and modifying the weight matrices accordingly. An attention layer in a transformer has query, key, value, and output projection matrices. With $H$ heads each of dimension $d_k$:

Each head $h$ uses columns $(h-1) \cdot d_k$ through $h \cdot d_k$ of these matrices. The output projection $\mathbf{W}_O \in \mathbb{R}^{(H \cdot d_k) \times d_{\text{model}}}$ assembles the contributions from all heads. To remove head $h$, you delete the corresponding columns from $\mathbf{Q}$, $\mathbf{K}$, $\mathbf{V}$, and the corresponding rows from $\mathbf{W}_O$.

After removing $k$ heads, the model dimension of the attention projections shrinks from $H \cdot d_k$ to $(H - k) \cdot d_k$. This directly reduces the memory footprint of the attention layer and the number of FLOPs in every forward pass. The change is structural and hardware-visible: the new, smaller weight matrices are stored in contiguous memory, and standard matrix multiplication routines operate on the smaller tensors with no overhead.

Pruning decisions can be made at the layer level (removing a fixed number of heads from each layer independently) or globally (ranking all heads across all layers together and removing the globally least important ones). Global pruning is generally preferable because it finds that some layers have more redundant heads than others. In BERT, for example, researchers consistently found that middle layers had more redundant heads than early or late layers. A layer-level strategy that forces uniform pruning across layers would retain unnecessary heads in middle layers while potentially over-pruning early or late layers.

Removing heads causes an immediate quality drop. The model was trained with all heads present, and the surviving heads were not optimized to compensate for the removed ones. A brief fine-tuning pass, sometimes called recovery fine-tuning, allows the remaining heads to adapt. In practice, this recovery phase needs far fewer steps than the original training because you are not learning from scratch but adjusting weights that already encode useful representations.

The amount of quality you can recover depends on how aggressively you pruned. Removing 10-20% of heads typically recovers fully within a small number of fine-tuning steps. Removing 50% or more often results in permanent quality degradation, though the model may still be useful for applications that tolerate some accuracy loss in exchange for speed. The recovery fine-tuning learning rate should be substantially lower than the original training learning rate to avoid disrupting the representations that the surviving heads have already learned.

One practical consideration is the order of operations. If you plan to prune both layers and heads, the recommended order is: compute layer importance and remove layers first, then compute head importance on the reduced model and remove heads, then fine-tune. This order matters because removing layers changes the distribution of attention patterns, which can shift head importance scores. Computing head importance on the original model and then removing both layers and heads simultaneously tends to produce worse results.

Layer pruning removes entire transformer blocks from the model. A transformer block includes the full self-attention sublayer, the feedforward sublayer, and their associated layer normalization layers. Removing a block reduces the number of sequential operations in every forward pass and eliminates all associated parameters. For a 12-layer BERT model, each layer represents roughly 8% of the total transformer parameters (excluding embeddings), so removing one layer provides approximately that fraction of total parameter reduction.

The computational benefit of layer removal is particularly significant for autoregressive generation. In generation, each new token requires a full forward pass through all transformer layers. Removing two layers from a 12-layer model reduces per-token generation time by roughly 17%, a meaningful speedup when you are generating hundreds of tokens per request.

The conceptually cleanest importance score for a layer measures its contribution to the residual stream. Modern transformers use residual connections: each block adds its output to the input before passing to the next block. You can measure how much a layer changes the representation by computing the cosine similarity between the input and output of that layer:

where:

• $\mathbf{h}^{(l)}(x)$ is the hidden state entering layer $l$ for input $x$
• $\mathbf{h}^{(l+1)}(x)$ is the hidden state leaving layer $l$
• $\cos(\cdot, \cdot)$ is cosine similarity, ranging from -1 to 1
• $N$ is the number of tokens evaluated

A layer with similarity close to 1.0 barely modifies the representation; it is essentially a near-identity transformation. Such layers are strong candidates for removal. A layer with similarity much less than 1.0 applies a substantial transformation and is important to retain. This metric is intuitive because it measures what matters to the residual stream: whether the layer changes the information flowing through the model.

This block influence (BI) criterion was investigated by several research groups studying large language models and consistently found that some middle layers have very high input-output similarity, suggesting they contribute little. The distribution of layer similarities is not uniform: early layers typically apply large transformations (encoding syntactic and positional structure into the embeddings), late layers also apply significant transformations (projecting representations toward output token probabilities), and some middle layers are nearly identity. This U-shaped distribution of layer importance has been replicated across model families from BERT to GPT to Llama.

Another approach uses gradient-based sensitivity. Compute the gradient of the task loss with respect to the output of each transformer block. Layers where the gradient magnitude is small contribute little to the final prediction and can be removed without large quality drops:

where:

• $S^{(l)}$ is the sensitivity score for layer $l$
• $\frac{\partial \mathcal{L}}{\partial \mathbf{h}^{(l+1)}}$ is the gradient of the task loss with respect to the layer's output hidden state
• $\|\cdot\|_F$ is the Frobenius norm, measuring the overall magnitude of the gradient signal

Layers with large sensitivity scores are critical to the model's predictions because backpropagation flows strongly through them. Layers with small scores carry little gradient and can be removed with minimal quality impact. The gradient-based score is computationally more expensive than the cosine similarity score because it requires a backward pass, but it is more directly connected to the training objective.

A more refined version of the cosine similarity criterion uses angular distance rather than cosine similarity directly. For each layer, you compute how much the hidden state rotates on average as it passes through:

where:

• $\theta^{(l)}$ is the angular distance (in radians) for layer $l$
• $\text{sim}^{(l)}$ is the mean cosine similarity between the input and output hidden states of layer $l$
• $\arccos$ converts cosine similarity to an angle, with 0 meaning no rotation and $\pi$ meaning a complete reversal

Layers with small $\theta^{(l)}$ (small rotation, near-identity) are pruned first. This angular perspective is useful because it is invariant to the scale of the hidden state norms, focusing purely on the geometric transformation. A layer that scales all hidden states by a constant factor but does not rotate them would have $\theta^{(l)} = 0$ and would be correctly identified as removable, whereas a norm-based criterion might incorrectly flag such a layer as active because its output magnitude differs from its input.

The angular distance criterion is the foundation of the ShortGPT method, introduced by Ma and colleagues in 2024. They applied it to Llama 2 and found that removing the 25% of layers with the smallest angular distances produced models that retained over 95% of the original performance on standard benchmarks while running substantially faster. The layers identified as most removable were consistently in the middle of the network, confirming the U-shaped importance profile observed in earlier work.

When you drop a transformer block, the layers before and after it are now adjacent. The hidden state that was computed by layer $l-1$ flows directly into layer $l+1$. If the removed layer was performing a useful but subtle transformation, the model can often compensate through fine-tuning because the surviving layers can adjust their parameters to approximately recover the removed layer's contribution.

Recovery from layer pruning benefits significantly from targeted fine-tuning where only the layers adjacent to the removed block are updated initially, then the whole model is fine-tuned at a lower learning rate. This staged approach stabilizes training and typically achieves better final quality than fine-tuning the entire pruned model from the start. The intuition is that the adjacent layers need to make the largest adjustments and giving them a focused fine-tuning phase before introducing global optimization pressure prevents conflicting gradient signals.

When removing multiple layers, the order of removal matters. Removing the single least-important layer first, then recomputing similarities on the resulting model, and repeating this process gives better results than selecting the $k$ least-important layers simultaneously and removing them all at once. The iterative approach allows the model to adapt after each removal, which changes the importance scores for the remaining layers. This iterative single-layer removal is more compute-intensive but consistently produces higher-quality pruned models.

A practical consideration when deploying layer-pruned models is that layer count affects KV-cache size in autoregressive models. Each transformer layer contributes entries to the KV cache for each token in the context. Removing two layers from a 32-layer model reduces KV-cache memory by 6.25% for long contexts, a meaningful saving in memory-constrained deployments.

Width pruning targets the internal dimensions of individual layers rather than removing entire heads or layers. In transformer feedforward sublayers, the input dimension $d_{\text{model}}$ is projected up to an intermediate dimension $d_{\text{ffn}}$ (often 4 times larger), passed through a nonlinearity, then projected back down. Width pruning removes neurons from this intermediate dimension, making the feedforward sublayer smaller while keeping the layer itself in the architecture.

Width pruning is more fine-grained than layer or head pruning, which allows you to tune the compression ratio with greater precision. Instead of choosing between keeping or removing an entire layer (which jumps the parameter count by a large discrete amount), you can remove neurons one at a time, finding the exact tradeoff between size and quality that meets your deployment constraints.

For the feedforward sublayer, each intermediate neuron receives input from the first linear layer and sends output through the second. Neuron $n$ in the feedforward sublayer of layer $l$ has activation $a_n^{(l)}(x)$ for input $x$. A natural importance score is the mean absolute activation:

where:

• $I_n^{(l)}$ is the importance score for neuron $n$ in feedforward sublayer $l$
• $a_n^{(l)}(x)$ is the activation value of neuron $n$ for input $x$ after the first linear projection and nonlinearity
• $\mathbb{E}_{x \sim \mathcal{D}}$ is the expectation over the calibration dataset $\mathcal{D}$

Neurons that are consistently near zero contribute little to the computation. A threshold $\tau$ determines which neurons to remove: all neurons with $I_n^{(l)} < \tau$ are eliminated.

The choice of nonlinearity affects how well this criterion works. With ReLU activations, neurons that are dead (always outputting zero) have zero importance by definition. With GELU or SiLU activations, which are smooth and only approach zero asymptotically, you need the mean absolute activation criterion to identify low-importance neurons because no neuron is ever exactly zero. In modern language models that use GELU or SiLU, mean absolute activation is the standard criterion for feedforward width pruning.

The mechanics of width pruning are similar to head pruning. The first feedforward projection $\mathbf{W}_1 \in \mathbb{R}^{d_{\text{model}} \times d_{\text{ffn}}}$ has its columns pruned, and the second projection $\mathbf{W}_2 \in \mathbb{R}^{d_{\text{ffn}} \times d_{\text{model}}}$ has its rows pruned correspondingly. Removing $k$ neurons shrinks both matrices and reduces FLOPs proportionally. The bias terms (if present) must also be updated consistently: the bias of the first layer loses $k$ entries, and if the second layer has a bias, it remains unchanged in dimension because it is added to the $d_{\text{model}}$-dimensional output rather than the intermediate dimension.

For models using gated linear units (such as the SwiGLU variant used in Llama), the feedforward sublayer has two parallel projections instead of one. Both projections must be pruned consistently: removing neuron $n$ requires removing column $n$ from both the gate projection and the value projection, and row $n$ from the output projection. The paired nature of gated feedforward layers means you need to be careful to prune both projections together.

Width pruning can also target the head dimension $d_k$ within attention heads. Rather than removing entire heads, you reduce the dimensionality of each head by eliminating less-important dimensions of the query, key, and value projections. This is a finer-grained form of pruning that allows more gradual size reduction.

For attention head $h$, the query projection $\mathbf{W}_Q^{(h)} \in \mathbb{R}^{d_{\text{model}} \times d_k}$ can be pruned by removing columns whose corresponding activations have low importance. The same columns must be removed from $\mathbf{W}_K^{(h)}$ and $\mathbf{W}_V^{(h)}$ to maintain dimensional consistency, and the corresponding rows from the output projection must also be removed. Note that the attention score computation divides by $\sqrt{d_k}$, so after reducing $d_k$, you must update this scaling factor accordingly.

Head dimension pruning is less commonly applied in practice than feedforward width pruning because the attention head dimensions are already relatively small (often 64 or 128), and reducing them further tends to hurt quality more than removing entire low-importance heads. Feedforward neurons are more numerous and more unevenly used, making them better targets for width pruning.

The most aggressive form of width pruning targets the model dimension $d_{\text{model}}$ itself, reducing the embedding size used throughout the entire model. This affects every weight matrix in the architecture simultaneously and requires careful identification of which dimensions to prune.

Principal component analysis on the activations at each layer can identify dimensions with low variance across inputs. These low-variance dimensions carry little information and are candidates for removal. You compute the covariance matrix of the activations across your calibration dataset, find the eigenvectors with the smallest eigenvalues, and remove the corresponding dimensions from all weight matrices simultaneously. This is related to the idea of intrinsic dimensionality: large language models often operate in a much lower-dimensional subspace than their nominal parameter count suggests.

Hidden dimension pruning is substantially harder to implement correctly than feedforward or head pruning because it requires coordinated changes to all weight matrices in the model. Every embedding matrix, every query, key, value, and output projection, every feedforward projection, and every layer normalization parameter must be updated. Small misalignments (for example, pruning the wrong corresponding rows and columns in two matrices) produce incorrect results without necessarily raising an obvious error. In practice, most practitioners use feedforward and head pruning rather than hidden dimension pruning because the former are safer to implement and still achieve substantial compression ratios.

To make these concepts concrete, consider a small BERT-like encoder with 6 layers, 8 attention heads per layer, a model dimension of 512, and a feedforward dimension of 2048. We will walk through each pruning strategy step by step with specific numbers.

Starting parameters:

The full model has approximately:

• Attention weights: $6 \times 4 \times 512^2 = 6{,}291{,}456$ parameters (Q, K, V, O projections per layer)
• Feedforward weights: $6 \times 2 \times (512 \times 2048) = 12{,}582{,}912$ parameters
• Total (excluding embeddings): approximately 18.9 million parameters

With $H = 8$ heads and $d_k = 64$ per head, each attention layer contains 4 projection matrices, each $512 \times 512 = 262{,}144$ parameters, for a total of about 1,048,576 per layer. The feedforward sublayer with $d_{\text{ffn}} = 2048$ has two projection matrices totalling $2 \times 512 \times 2048 = 2{,}097{,}152$ parameters per layer. Across 6 layers, the feedforward sublayers collectively dominate the parameter count.

Step 1: Compute layer similarities

Run 1000 representative inputs through the model and compute input-output cosine similarity for each of the 6 layers. Suppose you find:

Layers 3 and 4 have near-identity behavior and are candidates for removal. Notice that the high-similarity layers are not at the ends but in the middle, which is the typical pattern for BERT-style encoders. Removing layers 3 and 4 eliminates 2 of the 6 transformer blocks, reducing the transformer parameter count by 33% and cutting latency by roughly the same fraction (assuming attention and feedforward layers dominate computation, which is true for longer sequences).

Step 2: Head importance scoring

For each head in each of the remaining 4 layers (after considering layer removal), compute gradient-based importance scores. With 4 layers and 8 heads per layer, you have 32 heads in total. Suppose that roughly 6 heads across the 4 layers have importance scores more than one standard deviation below the mean. These 6 heads are candidates for removal, reducing the head count from 32 to 26 and shrinking the attention weight matrices proportionally.

The heads with the lowest importance scores are often concentrated in one or two specific layers. In the typical pattern, layers just after the removed layers (layer 5 in our example, which was previously layer 3 but is now adjacent to where layers 3 and 4 were removed) sometimes show reduced head importance because they had been receiving partially pre-processed representations from the removed layers and now receive less-processed input.

Step 3: Feedforward width assessment

For each feedforward sublayer in the remaining 4 layers, compute mean absolute activation for each of the 2048 intermediate neurons over your calibration set. If you find that roughly 20% of neurons have mean absolute activation below a threshold of 0.01, you can remove those 400 neurons per layer across 4 layers, totalling 1,600 neuron removals. Each removal eliminates one column from $\mathbf{W}_1$ and one row from $\mathbf{W}_2$ in that layer, proportionally reducing feedforward computation.

Step 4: Prune and recover

Remove layers 3 and 4 (leaving 4 layers), then remove the 6 low-importance heads from the 4 remaining layers, then remove the low-activation feedforward neurons. Fine-tune the resulting model for 5% of the original training steps with a learning rate 10 times smaller than the original. The resulting model has roughly:

• 4 transformer layers instead of 6: 33% reduction in layer count
• 26 heads across 4 layers instead of 32: 19% reduction in attention heads
• 80% of feedforward neurons retained: 20% reduction in feedforward width

The combined effect on parameters is multiplicative across the architectural dimensions, typically producing a model around 40-50% smaller than the original with quality loss of 1-3 points on standard benchmarks after recovery fine-tuning.

We implement structured pruning for a transformer model, focusing on head pruning and layer pruning with gradient-based importance scores.

We start by building a simple multi-layer transformer encoder that we can prune. This implementation makes head pruning explicit by treating each head independently.

The model starts with a fixed number of parameters spread across 6 transformer blocks, each with 8 attention heads. Even at this small scale, the structural relationships between layers and heads are the same as in production-scale models: each block contains an attention sublayer and a feedforward sublayer connected by residual connections.

We compute the cosine similarity between input and output for each layer. Layers where the hidden state barely changes are candidates for removal. The function below passes a batch through each block sequentially, saving the input before each block and comparing it to the output.

For head pruning, we use gradient-based importance scores. We run a forward pass to compute the loss, backpropagate to get gradients, and compute the importance of each head using the gradient-weight product norm. This is the Taylor expansion criterion: it approximates how much the loss would change if that head's output were zeroed out.

With layer similarities computed, we remove the least important layers by creating a new model with a reduced number of transformer blocks. The pruning function deep-copies the model to preserve the original, then filters out the blocks at the specified indices.

Removing layers that apply near-identity transformations eliminates their parameters while preserving the model's functional behavior as closely as possible. The parameter reduction is proportional to the number of layers removed divided by the total number of layers, which for two layers out of six is roughly 33%.

We implement head pruning by physically removing the head-corresponding slices from the attention weight matrices. The function below handles the index arithmetic carefully: each head occupies a contiguous slice of columns in the Q, K, V matrices and a contiguous slice of rows in the output projection.

The pruned model has fewer parameters in its attention layers, with the size reduction proportional to the number of heads removed. The global selection strategy means the removed heads are concentrated in layers where importance scores are most unevenly distributed, which is exactly the desired behavior.

In practice, you apply layer pruning and head pruning together to maximize compression while minimizing quality loss. The combined function first removes low-similarity layers, then recomputes head importance on the smaller model and removes low-importance heads.

The key parameters for structured pruning are:

• layer_sim_threshold: Cosine similarity threshold above which a layer is considered near-identity. Typical values range from 0.80 to 0.95. Setting this too low removes important layers; setting it too high removes nothing. Start at 0.90 and adjust based on observed quality on a validation set.
• head_prune_fraction: Fraction of total attention heads to remove globally. Start conservatively at 10-20% and increase only after verifying quality recovery at lower fractions.
• max_layers_to_remove: Upper bound on how many layers to prune in a single pass. Removing too many at once makes recovery harder. For 12-layer models, start with at most 2 layers per pruning round.
• recovery_steps: Number of fine-tuning steps after pruning. Typically 5-20% of original training steps, with a learning rate 5-10 times smaller than the original.

A single aggressive pruning pass often damages quality more than a series of smaller pruning steps with fine-tuning between each step. Iterative pruning cycles through: prune a small amount, fine-tune to recover, evaluate quality, and repeat until the target size is reached. Each fine-tuning phase allows the model to stabilize before the next round of pruning begins. The accumulated effect of many small, recoverable pruning steps is a final model that is both smaller and higher quality than you would get from one large pruning step.

The intuition behind iterative pruning is that importance scores change after each pruning round. When you remove a low-importance head, the remaining heads may redistribute their roles. A head that appeared unimportant under the original model may become more important when another head in the same layer is removed, because the remaining head must now cover both roles. Iterative pruning updates importance scores between rounds, catching these redistributions and making better decisions.

The pruning schedule can be described as a function that determines what fraction of the model to prune at each step $t$:

where:

• $s_t$ is the sparsity at step $t$
• $s_i$ is the initial sparsity (often 0)
• $s_f$ is the final target sparsity
• $t_0$ is the step at which pruning begins
• $n$ is the number of pruning steps
• $\Delta t$ is the interval between pruning steps

The cubic term $\left(1 - \frac{t - t_0}{n \cdot \Delta t}\right)^3$ is the key feature of this schedule. At the beginning of pruning (when $t$ is just above $t_0$), the term is close to 1, so $s_t$ is close to $s_i$: pruning starts slowly. As $t$ increases toward $t_0 + n \cdot \Delta t$, the term approaches 0 and $s_t$ approaches $s_f$: pruning accelerates toward the target. Near the end of the schedule, as the target approaches, the cubic function causes pruning to decelerate again, making fine adjustments rather than large jumps. This S-shaped progression (slow, fast, slow) is better matched to how models learn than a constant linear rate.

The cubic schedule starts pruning slowly, accelerates through the middle, then slows down again near the target. This allows the model to accommodate each wave of removals before more components are pruned. The deceleration near the target is particularly important: the final few percentage points of sparsity are often the hardest to achieve without quality loss, and the schedule naturally allocates more fine-tuning time near the target.

The chart below compares the cubic schedule to a linear schedule. The cubic schedule removes fewer components early in training, when the model is still adjusting from prior fine-tuning, and reaches the target sparsity more gradually.

All importance-based pruning methods require a calibration dataset: a representative set of inputs used to compute activation statistics, gradients, or similarity scores. The quality of pruning decisions depends directly on how representative this calibration set is.

For a deployed sentiment classifier, you should calibrate using typical user inputs from production traffic rather than the original training data. The distribution of production traffic often differs from the training distribution in ways that affect which components are important. For a question-answering model, the calibration set should include questions across diverse domains, because heads and layers that handle medical questions may differ from those that handle programming questions.

Using an unrepresentative calibration set leads to pruning decisions optimized for the wrong inputs. A model calibrated on formal text may incorrectly identify heads important for informal language as low-importance, leading to quality degradation on real user queries. A model calibrated on short sentences may retain heads that primarily contribute to long-range dependencies, which are underutilized on short inputs but critical for longer ones.

In practice, 1,000 to 10,000 representative inputs provide stable importance estimates for most models. Using far fewer (under 100) leads to noisy estimates and worse pruning decisions, because the sample size is too small to average out example-specific variation in which heads activate. Using more than 10,000 inputs rarely improves importance score quality but increases the computational cost of calibration linearly.

The calibration set should not be the same as the fine-tuning dataset used for recovery. Using the same data for both calibration (to compute importance scores) and recovery fine-tuning (to restore quality) can lead to overfitting, where the pruned model becomes optimized specifically for that dataset rather than the broader task distribution. A practical recommendation is to split your available labeled data three ways: a calibration set for importance scoring (roughly 10-20%), a fine-tuning set for recovery (roughly 70-80%), and a validation set for measuring quality throughout the pruning process (roughly 10-20%).

Several production systems have adopted structured pruning as part of their model deployment pipeline. The patterns that have emerged from these deployments provide practical guidance that goes beyond what the research literature covers.

Task-specific pruning is more effective than generic pruning. When you know your target task, you can calibrate on task-relevant data and fine-tune specifically for that task, recovering more quality at the same compression ratio than you would achieve with task-agnostic pruning. A BERT model pruned for named entity recognition on biomedical text can be much more aggressively compressed than a general-purpose BERT model, because many of the model's general linguistic capabilities are irrelevant for that specific task.

Pruning interacts with other compression techniques. Quantization applied after pruning compounds the savings. A model that is 40% smaller from structured pruning and then quantized to 8-bit precision can be 70-80% smaller than the original full-precision model. The quality loss from quantization tends to be smaller when applied to a pruned model, because the pruned model has already been fine-tuned to work well with fewer components. Applying distillation after pruning is also effective: you prune first to get the target architecture, then use the original model as a teacher to improve the quality of the pruned model beyond what recovery fine-tuning alone achieves.

Early detection of pruning failures saves time. After each pruning round, evaluate on your validation set before proceeding to the next round or starting full recovery fine-tuning. If the quality drop after pruning is much larger than expected (more than 10-15 points on typical benchmarks for mild pruning), it usually indicates that you have removed a critical component. Stop and diagnose before investing compute in recovery fine-tuning that is unlikely to succeed.

Not all tasks prune equally well. Tasks that require broad linguistic knowledge (such as reading comprehension across diverse domains) tend to prune less gracefully than narrow tasks (such as binary sentiment classification on product reviews). The former relies on many components working together; the latter uses a small subset of the model's capabilities and can tolerate removing large parts of the rest.

Structured pruning delivers real speedups but comes with significant practical challenges that affect how aggressively you can prune in production settings.

The most fundamental limitation is the accuracy-efficiency tradeoff. Every component you remove carries some information that the model was using, even if the component was relatively unimportant. With mild pruning (removing 10-20% of heads or 1-2 layers from a 12-layer model), quality usually recovers fully after fine-tuning. With aggressive pruning (removing 40-50% of components), some permanent quality degradation is unavoidable. The degree of degradation depends on the task: tasks that require broad linguistic knowledge degrade more than narrow tasks that use only a subset of the model's capabilities.

Recovery fine-tuning requires labeled data for the downstream task. This is straightforward for supervised classification tasks but harder for generative models, where you need a proxy objective that correlates with generation quality. Using next-token prediction loss on a representative text corpus works for language models, but the quality of the pruned generative model may not improve as smoothly as for classifiers. For generative models, human evaluation or automatic metrics like ROUGE or BERTScore should be used to track quality throughout the pruning process, not just training loss.

The importance score computation itself has limitations. Gradient-based importance scores are computed on a specific calibration dataset and may not generalize across domains. A head that is unimportant for formal news text may be necessary for informal conversational text. This means pruning decisions made for one deployment context may not transfer well to other contexts, requiring re-pruning when the use case changes.

Structured pruning interacts with other compression techniques in important ways. Applying quantization after pruning compounds the compression but also compounds potential quality loss. Applying distillation after pruning is more stable: you prune first to get the target architecture, then distill to recover quality from a larger teacher. This two-stage pipeline (prune then distill) often outperforms either technique applied alone.

Head pruning has an interesting failure mode: when you remove an attention head, the surviving heads may not fully compensate because their query and key spaces were trained independently. Multi-head attention does not guarantee that any single head can replicate the behavior of another. Recovery fine-tuning helps, but there are limits to how much compensation is possible without retraining from scratch.

Layer pruning has a different failure mode: removing middle layers can disrupt the flow of information from early to late layers. If removed layers were responsible for building up specific intermediate representations that later layers relied on, the resulting model cannot route information correctly even after fine-tuning. Careful importance scoring based on layer similarity helps identify which layers are safe to remove, but abrupt removal of critical layers can be difficult to recover from. This is why the iterative approach (remove one layer at a time with fine-tuning between rounds) is more reliable than removing multiple layers simultaneously.

Finally, structured pruning is sensitive to the model's pretraining history. Models that were pretrained with larger batches and for more steps tend to be more amenable to pruning, because they have had more opportunity to develop redundancy in their components. Models that were undertrained relative to their size (common in resource-constrained academic settings) often cannot be pruned as aggressively because their components are not as redundant. When working with a model of unknown provenance, start with a conservative pruning ratio and increase gradually while monitoring quality.

Structured pruning removes entire architectural components from transformers, producing models that run faster on any hardware without requiring specialized sparse computation support.

• Head pruning removes entire attention heads using importance scores based on gradient magnitudes, activation norms, or loss sensitivity. After removal, the attention projection matrices shrink and every forward pass uses fewer FLOPs. Global pruning across all layers outperforms layer-by-layer pruning because it identifies layers with more redundant heads.
• Layer pruning removes complete transformer blocks. The cosine similarity between a block's input and output identifies near-identity layers that contribute little. Middle layers are often the best candidates, following the U-shaped importance profile observed across BERT, GPT, and Llama model families. Angular distance (via $\arccos$) provides a scale-invariant version of this criterion.
• Width pruning reduces the internal dimensions of feedforward sublayers or attention head dimensions by eliminating low-activation neurons. This offers fine-grained control over model size that complements the coarser-grained head and layer pruning strategies. Gated feedforward layers (SwiGLU) require coordinated pruning of both the gate and value projections.
• Iterative pruning with recovery fine-tuning between rounds achieves better final quality than a single aggressive pruning pass. Cubic or polynomial schedules balance compression speed and quality retention by allowing the model more adaptation time at the critical early and late stages of the pruning process.
• Calibration data quality determines how accurate importance scores are. Use representative data from the actual deployment context, and keep calibration and fine-tuning data separate to avoid overfitting.
• Combining techniques amplifies compression ratios. Structured pruning reduces architecture size, quantization reduces numerical precision, and distillation transfers knowledge from a larger model. Together, these techniques make large language models deployable in contexts where the original models would be impractical.

Structured pruning is most powerful when combined with other compression techniques. Pruning reduces architecture size, quantization reduces numerical precision, and distillation transfers knowledge from a larger model. Together, these techniques make large language models deployable in contexts where the original models would be impractical. The next part of this handbook explores reasoning capabilities in language models, where we will see how efficient inference from compressed models enables deploying chain-of-thought reasoning in constrained environments.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about structured pruning.