Structured Pruning › Width Pruning

Width pruning targets the internal dimensions of individual layers rather than removing entire heads or layers. In transformer feedforward sublayers, the input dimension $d_{\text{model}}$ is projected up to an intermediate dimension $d_{\text{ffn}}$ (often 4 times larger), passed through a nonlinearity, then projected back down. Width pruning removes neurons from this intermediate dimension, making the feedforward sublayer smaller while keeping the layer itself in the architecture.

Width pruning is more fine-grained than layer or head pruning, which allows you to tune the compression ratio with greater precision. Instead of choosing between keeping or removing an entire layer (which jumps the parameter count by a large discrete amount), you can remove neurons one at a time, finding the exact tradeoff between size and quality that meets your deployment constraints.

For the feedforward sublayer, each intermediate neuron receives input from the first linear layer and sends output through the second. Neuron $n$ in the feedforward sublayer of layer $l$ has activation $a_n^{(l)}(x)$ for input $x$. A natural importance score is the mean absolute activation:

$$I_n^{(l)} = \mathbb{E}_{x \sim \mathcal{D}} \left[ \left| a_n^{(l)}(x) \right| \right]$$

where:

• $I_n^{(l)}$ is the importance score for neuron $n$ in feedforward sublayer $l$
• $a_n^{(l)}(x)$ is the activation value of neuron $n$ for input $x$ after the first linear projection and nonlinearity
• $\mathbb{E}_{x \sim \mathcal{D}}$ is the expectation over the calibration dataset $\mathcal{D}$

Neurons that are consistently near zero contribute little to the computation. A threshold $\tau$ determines which neurons to remove: all neurons with $I_n^{(l)} < \tau$ are eliminated.

The choice of nonlinearity affects how well this criterion works. With ReLU activations, neurons that are dead (always outputting zero) have zero importance by definition. With GELU or SiLU activations, which are smooth and only approach zero asymptotically, you need the mean absolute activation criterion to identify low-importance neurons because no neuron is ever exactly zero. In modern language models that use GELU or SiLU, mean absolute activation is the standard criterion for feedforward width pruning.

The mechanics of width pruning are similar to head pruning. The first feedforward projection $\mathbf{W}_1 \in \mathbb{R}^{d_{\text{model}} \times d_{\text{ffn}}}$ has its columns pruned, and the second projection $\mathbf{W}_2 \in \mathbb{R}^{d_{\text{ffn}} \times d_{\text{model}}}$ has its rows pruned correspondingly. Removing $k$ neurons shrinks both matrices and reduces FLOPs proportionally. The bias terms (if present) must also be updated consistently: the bias of the first layer loses $k$ entries, and if the second layer has a bias, it remains unchanged in dimension because it is added to the $d_{\text{model}}$-dimensional output rather than the intermediate dimension.

For models using gated linear units (such as the SwiGLU variant used in Llama), the feedforward sublayer has two parallel projections instead of one. Both projections must be pruned consistently: removing neuron $n$ requires removing column $n$ from both the gate projection and the value projection, and row $n$ from the output projection. The paired nature of gated feedforward layers means you need to be careful to prune both projections together.

Width pruning can also target the head dimension $d_k$ within attention heads. Rather than removing entire heads, you reduce the dimensionality of each head by eliminating less-important dimensions of the query, key, and value projections. This is a finer-grained form of pruning that allows more gradual size reduction.

For attention head $h$, the query projection $\mathbf{W}_Q^{(h)} \in \mathbb{R}^{d_{\text{model}} \times d_k}$ can be pruned by removing columns whose corresponding activations have low importance. The same columns must be removed from $\mathbf{W}_K^{(h)}$ and $\mathbf{W}_V^{(h)}$ to maintain dimensional consistency, and the corresponding rows from the output projection must also be removed. Note that the attention score computation divides by $\sqrt{d_k}$, so after reducing $d_k$, you must update this scaling factor accordingly.

Head dimension pruning is less commonly applied in practice than feedforward width pruning because the attention head dimensions are already relatively small (often 64 or 128), and reducing them further tends to hurt quality more than removing entire low-importance heads. Feedforward neurons are more numerous and more unevenly used, making them better targets for width pruning.

The most aggressive form of width pruning targets the model dimension $d_{\text{model}}$ itself, reducing the embedding size used throughout the entire model. This affects every weight matrix in the architecture simultaneously and requires careful identification of which dimensions to prune.

Principal component analysis on the activations at each layer can identify dimensions with low variance across inputs. These low-variance dimensions carry little information and are candidates for removal. You compute the covariance matrix of the activations across your calibration dataset, find the eigenvectors with the smallest eigenvalues, and remove the corresponding dimensions from all weight matrices simultaneously. This is related to the idea of intrinsic dimensionality: large language models often operate in a much lower-dimensional subspace than their nominal parameter count suggests.

Hidden dimension pruning is substantially harder to implement correctly than feedforward or head pruning because it requires coordinated changes to all weight matrices in the model. Every embedding matrix, every query, key, value, and output projection, every feedforward projection, and every layer normalization parameter must be updated. Small misalignments (for example, pruning the wrong corresponding rows and columns in two matrices) produce incorrect results without necessarily raising an obvious error. In practice, most practitioners use feedforward and head pruning rather than hidden dimension pruning because the former are safer to implement and still achieve substantial compression ratios.