Pruning Basics › Worked Example: Magnitude Pruning by Hand

To make these concepts concrete, consider a small fully connected layer with a $3 \times 4$ weight matrix:

$$W = \begin{bmatrix} 0.8 & -0.02 & 0.5 & 0.01 \\ -0.3 & 0.7 & -0.01 & 0.6 \\ 0.05 & -0.4 & 0.9 & -0.03 \end{bmatrix}$$

This matrix has 12 weights total, connecting 4 input neurons to 3 output neurons. We want to apply 50% unstructured magnitude pruning, meaning we will zero out the 6 weights with the smallest absolute values.

The absolute values in ascending order are: 0.01, 0.01, 0.02, 0.03, 0.05, 0.05, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9. The threshold at 50% sparsity falls at $|w| = 0.05$ (the 6th smallest value). All weights with $|w_{ij}| \leq 0.05$ are zeroed. The surviving weights form the pruned matrix:

$$W_{\text{pruned}} = \begin{bmatrix} 0.8 & 0 & 0.5 & 0 \\ -0.3 & 0.7 & 0 & 0.6 \\ 0 & -0.4 & 0.9 & 0 \end{bmatrix}$$

Six out of twelve weights are now exactly zero. The matrix has the same shape as before but contains 50% zeros. In a sparse storage format like CSR, we store only 6 nonzero values plus their column indices, roughly halving the storage requirement. On hardware that supports sparse computation, the matrix-vector product $W_{\text{pruned}} \mathbf{x}$ can skip the 6 zero multiplications.

Notice that the pruning preserved the structure of the large weights. The values 0.8, 0.7, 0.9, 0.5, 0.6, and 0.4 all survived, while the cluster of near-zero values (0.01, 0.02, 0.03, etc.) were removed. The large weights are likely encoding the most important signal, and the small weights are noise.

For structured pruning on the same matrix, we would instead consider removing entire rows, where each row corresponds to one output neuron. The importance score for each row is its L2 norm:

• Row 1: $\sqrt{0.8^2 + 0.02^2 + 0.5^2 + 0.01^2} \approx \sqrt{0.64 + 0.0004 + 0.25 + 0.0001} \approx 0.94$
• Row 2: $\sqrt{0.3^2 + 0.7^2 + 0.01^2 + 0.6^2} \approx \sqrt{0.09 + 0.49 + 0.0001 + 0.36} \approx 0.96$
• Row 3: $\sqrt{0.05^2 + 0.4^2 + 0.9^2 + 0.03^2} \approx \sqrt{0.0025 + 0.16 + 0.81 + 0.0009} \approx 0.99$

To prune 33% of neurons (removing 1 of 3 rows), we remove Row 1, which has the smallest norm. The result is a $2 \times 4$ weight matrix, not a $3 \times 4$ matrix with a zero row. This reduces the network's hidden dimension. Any matrix multiplications downstream of this layer also shrink, because the output dimension of this layer drops from 3 to 2. The result is a model that runs faster on any hardware.

The contrast between the two pruned matrices illustrates the core tradeoff. The unstructured-pruned $3 \times 4$ matrix and the structured-pruned $2 \times 4$ matrix both represent roughly the same amount of removed information, but only the structured-pruned matrix produces a smaller computation graph.