Prompt Tuning › Summary

Prompt tuning simplifies parameter-efficient fine-tuning to its logical minimum: prepend learnable soft prompt embeddings to the input and freeze everything else. Unlike prefix tuning, which modifies attention computations at every layer, prompt tuning operates only at the input level, trusting the frozen model to propagate task-needed information through its layers using its own mechanisms.

The key takeaways from this chapter are:

• Formulation: Soft prompts $\mathbf{P} \in \mathbb{R}^{p \times d}$ are concatenated with input embeddings to form $\mathbf{X}' = [\mathbf{P}; \mathbf{X}]$, which is processed by the frozen model $f_\theta$ with no architectural changes. Only $\mathbf{P}$ receives gradient updates during training, with the total trainable parameter count being simply $p \times d$.

• Initialization matters at small scales: Vocabulary initialization (sampling starting points from existing token embeddings) outperforms random initialization, especially for smaller models. Class-label initialization gives even stronger inductive bias for classification tasks. At scales above approximately 1 billion parameters, all strategies converge to similar final performance.

• Scaling behavior defines applicability: The performance gap between prompt tuning and full fine-tuning shrinks reliably with model scale, reaching parity at approximately 10 billion parameters. This makes prompt tuning increasingly attractive for very large models while alternatives like LoRA or prefix tuning may be preferred for smaller models.

• Prompt length controls capacity versus context: Moderate prompt lengths (10-100 tokens) typically work best, with the optimal value depending on task complexity and available context window. Each soft prompt token consumes one position from the context window, creating a trade-off that prefix tuning avoids.

• Limitations are real and practical: Scale dependence, soft prompt interpretability challenges, context window consumption, and optimization sensitivity are constraints that determine when prompt tuning is the right choice versus a full fine-tuning or alternative PEFT approach.

The parameter efficiency of prompt tuning is remarkable: adapting a 10 billion parameter model may require training only 100,000 parameters, roughly 0.001% of the total. This makes possible maintaining hundreds of task-specific adaptations in the space of a single additional model copy, transforming the economics of multi-task model deployment. Organizations serving many distinct tasks can maintain a single high-quality base model and a library of tiny prompt matrices, rather than managing dozens of full fine-tuned model copies.

The next chapter explores adapter layers, a different approach that inserts small trainable bottleneck modules between transformer layers. Adapters give an alternative intervention point that avoids the context window constraint of prompt tuning while remaining more parameter-efficient than prefix tuning at intermediate model scales. Subsequent chapters then compare all PEFT methods systematically to guide your choices across the full space of model scales, task types, and deployment constraints.