Structured Pruning › Calibration Data Requirements

All importance-based pruning methods require a calibration dataset: a representative set of inputs used to compute activation statistics, gradients, or similarity scores. The quality of pruning decisions depends directly on how representative this calibration set is.

For a deployed sentiment classifier, you should calibrate using typical user inputs from production traffic rather than the original training data. The distribution of production traffic often differs from the training distribution in ways that affect which components are important. For a question-answering model, the calibration set should include questions across diverse domains, because heads and layers that handle medical questions may differ from those that handle programming questions.

Using an unrepresentative calibration set leads to pruning decisions optimized for the wrong inputs. A model calibrated on formal text may incorrectly identify heads important for informal language as low-importance, leading to quality degradation on real user queries. A model calibrated on short sentences may retain heads that primarily contribute to long-range dependencies, which are underutilized on short inputs but critical for longer ones.

In practice, 1,000 to 10,000 representative inputs provide stable importance estimates for most models. Using far fewer (under 100) leads to noisy estimates and worse pruning decisions, because the sample size is too small to average out example-specific variation in which heads activate. Using more than 10,000 inputs rarely improves importance score quality but increases the computational cost of calibration linearly.

The calibration set should not be the same as the fine-tuning dataset used for recovery. Using the same data for both calibration (to compute importance scores) and recovery fine-tuning (to restore quality) can lead to overfitting, where the pruned model becomes optimized specifically for that dataset rather than the broader task distribution. A practical recommendation is to split your available labeled data three ways: a calibration set for importance scoring (roughly 10-20%), a fine-tuning set for recovery (roughly 70-80%), and a validation set for measuring quality throughout the pruning process (roughly 10-20%).