Prefix Tuning › Key Parameters

Understanding the key hyperparameters of prefix tuning helps you configure it effectively for your specific use case. Each parameter controls a different aspect of the capacity-efficiency tradeoff.

The prefix length ($m$) is the most important hyperparameter, controlling the number of virtual tokens added to each layer's key-value sequences. Longer prefixes increase steering capacity but consume attention budget and risk overfitting on small datasets. The right value depends primarily on task complexity, as described in the prefix length selection section. A good starting range is 20-50 for most tasks.

The prefix hidden dimension ($d'$) is the dimension of the compressed representation used by the reparameterization MLP. A smaller value gives more regularization and reduces training parameters, at the cost of potentially insufficient expressivity. A larger value allows the MLP more freedom to generate diverse prefix patterns but increases the risk of overfitting. Values in the range 256-1024 typically work well; setting it equal to the model's hidden dimension is a common choice.

The learning rate applied to prefix encoder parameters is often higher than what you would use for full fine-tuning, because the prefix encoder starts from a random initialization and has many fewer parameters to update. Learning rates in the range $1 \times 10^{-3}$ to $5 \times 10^{-4}$ are common starting points, compared to $1 \times 10^{-5}$ for full fine-tuning of a similar model.

The number of training epochs may also differ from full fine-tuning. Because only a tiny fraction of parameters are trainable, each update makes a relatively large proportional change to those parameters. Overfitting can occur after fewer epochs than with full fine-tuning, particularly on small datasets. Monitoring validation loss carefully and using early stopping is especially important.