Prefix Tuning › Summary

Prefix tuning gives a parameter-efficient alternative to full fine-tuning by prepending learnable continuous vectors to the key and value sequences in attention. Rather than modifying model weights like LoRA, it changes what the attention mechanism sees, steering behavior through persistent virtual context that influences every layer at every step.

The key concepts from this chapter are:

• Formulation: Prefix tuning concatenates learnable matrices $P_K$ and $P_V$ to the key and value sequences at each transformer layer, extending the attention scope without changing any model weights. The query sequences remain untouched, preserving output sequence length.

• Layer specificity: Separate prefix matrices are learned for each transformer layer, letting task-specific steering at every level of the representational hierarchy, from basic syntactic processing in early layers to high-level semantic control in later layers.

• Reparameterization: Training stability improves materially by generating all layer-specific prefixes through a shared MLP rather than optimizing them directly. This bottleneck prevents attention saturation and smooths the optimization objective, and the MLP can be discarded after training.

• Prefix length: The primary capacity hyperparameter, typically ranging from 10-200 depending on task complexity. Simple classification benefits from shorter prefixes while complex generation tasks may require longer ones. The cost of each additional prefix token is a small but real fraction of the attention budget.

• Generation focus: Prefix tuning excels at controlling generation style and format. This gives consistent task guidance throughout autoregressive decoding without consuming context window space or diminishing in influence over long generations.

• Comparison with LoRA: While LoRA modifies weights and can be merged for zero-overhead inference, prefix tuning modifies inputs and requires maintaining separate prefix vectors. LoRA tends to excel at discriminative tasks while prefix tuning is stronger for generation tasks requiring consistent behavioral guidance.

The method reaches competitive performance with a small fraction of base model parameters, which makes it practical for adapting large language models to specific tasks without the computational infrastructure required for full fine-tuning. For generation tasks requiring consistent style or format, prefix tuning offers an elegant solution that keeps the base model completely frozen while giving powerful, persistent, task-specific guidance at every layer of the transformer hierarchy.