Prompt Tuning › From Prefix Tuning to Prompt Tuning

As we discussed in the previous chapter, prefix tuning reaches parameter efficiency by prepending trainable key-value pairs to each transformer layer. This design shows the observation that attention patterns throughout the network could be steered by modifying what the model attends to at every layer. The prefix acts as a continuous source of advice that every attention head in every layer can consult, directly shaping the computation at the point where it matters most.

However, this layer-wise approach has a cost: the number of parameters scales linearly with both the prompt length and the number of layers. A model with 96 layers and a prefix of 100 tokens requires parameters for each layer multiplied by the prefix length. For the prefix key and value matrices combined, this means roughly $100 \times 96 \times 2 \times d$ parameters, which becomes substantial for large embedding dimensions. The reparameterization trick (using an MLP to generate the actual prefix embeddings from a smaller matrix) reduces optimization instability but adds another component to maintain and tune.

Prompt tuning takes a fundamentally different approach. Instead of intervening at every layer, it intervenes only at the input, inserting learnable virtual tokens before the actual input tokens. The frozen transformer then propagates task-specific information from these input embeddings through all subsequent layers using its own attention and feed-forward computations. The model's pretrained capabilities handle task adaptation internally, without any layer-specific guidance.

Think of the difference like this: prefix tuning is like having a knowledgeable advisor whispering guidance into a person's ear at every decision they make throughout the day. Prompt tuning is like giving that person a detailed briefing note at the start of the day and trusting them to remember and apply the guidance on their own. The first approach guarantees the guidance reaches every decision point; the second relies on the person's capacity to internalize and apply the briefing throughout the day. For people with a great memory and broad expertise (large models), the briefing approach works just as well and is much simpler to arrange.

This design choice has several implications that cascade through the entire method. First, the parameter count becomes independent of model depth. A 12-layer model and a 96-layer model require identical numbers of trainable parameters for the same prompt length and embedding dimension. Second, the approach is architecturally minimal because there are no modifications to any transformer layer, no hooks into attention computations, and no new intermediate representations to manage. Third, prompt tuning's behavior depends entirely on the model's ability to use the soft prompt context effectively, which turns out to be strongly correlated with model scale.

The key insight driving prompt tuning is that large language models have learned such rich internal representations during pretraining that a small modification at the input level can cascade through the network to produce task-appropriate outputs. The model has seen billions of tokens in pretraining that taught it to recognize context and adapt its processing accordingly. Soft prompts use this existing capacity by learning to activate the right internal behaviors from the very first token the model sees. The success of this approach thus depends critically on model scale, as we will examine in detail when discussing scaling behavior.