Prompt Tuning › Prompt Tuning Formulation

To understand prompt tuning mathematically, we need to understand how soft prompts fit into the standard transformer input pipeline. In a typical transformer model, input text first passes through a tokenizer that splits it into subword pieces, then each piece is mapped to a dense vector through an embedding lookup table. The result is a matrix of token embeddings, one row per token, that is the model's input. Prompt tuning intervenes at exactly this junction, between tokenization and the first transformer layer, by inserting additional learnable vectors.

The intervening mechanism is elegantly simple. We define a prompt matrix of learnable parameters, concatenate it with the input embedding matrix, and pass this combined sequence to the frozen transformer. From the model's perspective, it simply receives a slightly longer input sequence than usual, with the first few tokens being the soft prompt virtual tokens. No architectural modification is needed, no special handling is required, and the model processes these additional tokens with its standard attention and feed-forward computations.

Let $\mathbf{X} \in \mathbb{R}^{n \times d}$ stand for the embedded input sequence. Here $n$ is the sequence length (number of input tokens) and $d$ is the embedding dimension (the width of each token representation). Each row $\mathbf{X}_i$ is a $d$-dimensional vector that encodes both the token's meaning and its position in the sequence through position embeddings added during preprocessing.

Prompt tuning introduces a trainable prompt matrix $\mathbf{P} \in \mathbb{R}^{p \times d}$, where $p$ is the number of soft prompt tokens. Each row of this matrix is one learnable "virtual token" that exists purely in continuous embedding space. Unlike real tokens, there is no vocabulary index associated with these rows. They are free parameters that gradient descent can move anywhere in the $d$-dimensional space.

We form the augmented input by prepending the soft prompt matrix to the original input sequence:

$$\mathbf{X}' = [\mathbf{P}; \mathbf{X}] \in \mathbb{R}^{(p+n) \times d}$$

where:

• $\mathbf{X}'$: the augmented input matrix containing both the soft prompt tokens and the actual data
• $\mathbf{P} \in \mathbb{R}^{p \times d}$: the matrix of learnable soft prompt parameters, with each row being one virtual token
• $\mathbf{X} \in \mathbb{R}^{n \times d}$: the original input embedding matrix, obtained from the frozen embedding lookup table
• $[\cdot; \cdot]$: row-wise concatenation along the sequence length dimension, creating a matrix with $p + n$ rows

The concatenation stacks the prompt rows on top of the input rows, creating a new sequence that begins with $p$ soft prompt tokens followed by the $n$ actual input tokens. From the transformer's perspective, this augmented sequence looks just like any standard embedded input, except that the first $p$ positions contain optimized free parameters rather than embeddings retrieved from the vocabulary. This is the core reason prompt tuning requires no architectural modifications: it piggybacks entirely on the model's existing input processing infrastructure.

This concatenated sequence is then processed by the frozen transformer. If we denote the frozen language model as $f_\theta$ (where $\theta$ is the fixed pretrained weights), the output representations are:

$$\mathbf{Y} = f_\theta([\mathbf{P}; \mathbf{X}])$$

where:

• $\mathbf{Y} \in \mathbb{R}^{(p+n) \times d}$: the output representations from the final transformer layer, one row per input position
• $f_\theta$: the frozen transformer function, with all parameters $\theta$ held constant during training
• $[\mathbf{P}; \mathbf{X}]$: the concatenated sequence forming the effective model input

During training, only $\mathbf{P}$ receives gradient updates. The entire transformer with parameters $\theta$ remains frozen, meaning that backpropagation computes gradients with respect to the prompt parameters but treats all model weights as constants. This is a necessary implementation detail: when you compute the backward pass, gradients flow from the loss through the frozen transformer back to $\mathbf{P}$, but they do not accumulate for any transformer weight. The training objective is the standard task loss: cross-entropy for classification, sequence-to-sequence loss for generation, or any other appropriate objective.

Why does this formula make sense? Notice that the transformer's attention mechanism allows every token to attend to every other token in the sequence, subject to any masking. This means that when the model processes the actual input tokens at positions $p+1$ through $p+n$, they can freely attend to the soft prompt tokens at positions $1$ through $p$. The soft prompts function as a kind of "context header" that shapes how the model processes the real input. Through training, the soft prompts learn to give whatever contextual framing maximally helps the model perform the target task, regardless of what discrete human-readable instructions would say.

Prompt tuning's simplicity is most evident when we examine its parameter count precisely. Since soft prompts exist only at the input level, the number of trainable parameters depends solely on the number of virtual tokens and the embedding dimension. We compute this directly:

$$N_{\text{params}} = p \times d$$

where:

• $N_{\text{params}}$: the total number of trainable parameters in the soft prompt
• $p$: the number of soft prompt tokens (the primary hyperparameter you control)
• $d$: the embedding dimension of the model (fixed by the pretrained architecture)

To make this concrete, consider a model with embedding dimension $d = 1024$ and a prompt length of $p = 100$ tokens. This setup yields exactly $100 \times 1024 = 102,400$ trainable parameters. For a model with one billion total parameters, this is approximately 0.01% of parameters receiving gradient updates. The remaining 99.99% are completely frozen throughout training.

The comparison with prefix tuning is equally striking and reveals how much prefix tuning's layer-wise design costs in parameters. Prefix tuning requires parameters proportional to $p \times L \times 2 \times d$, where $L$ is the number of transformer layers and the factor of 2 accounts for both key and value prefixes at each layer. For a 24-layer model with embedding dimension $d = 1024$ and prompt length $p = 100$, prefix tuning needs:

$$N_{\text{prefix}} = 100 \times 24 \times 2 \times 1024 = 4{,}915{,}200 \text{ parameters}$$

compared to prompt tuning's 102,400 parameters: a 48-fold difference. For a 96-layer model, the difference grows to 192-fold. Prompt tuning's parameter count is completely independent of model depth, which makes it more attractive precisely as models grow deeper and more powerful.

The formulation reveals a key architectural difference from prefix tuning that changes the interpretation for how each method steers the model's behavior. In prefix tuning, the soft prefixes directly modify the attention computation at every layer by adding to the keys and values that every attention head sees:

$$\text{Attention}(\mathbf{Q}, [\mathbf{P}_K; \mathbf{K}], [\mathbf{P}_V; \mathbf{V}])$$

where:

• $\mathbf{Q}$: the query matrix derived from the current layer's input
• $\mathbf{P}_K, \mathbf{P}_V \in \mathbb{R}^{p \times d_k}$: the trainable prefix matrices for keys and values at this layer
• $\mathbf{K}, \mathbf{V}$: the key and value matrices derived from the input at this layer
• $[\cdot; \cdot]$: concatenation extending the key and value sequences with the prefix

This direct injection means that every attention head at every layer has access to learned key-value pairs that can guide its computations. The prefixes effectively give steering information exactly where the model makes its attention-based decisions. They augment the information available to the attention mechanism at each layer independently, giving each layer its own dedicated learned context. The prefix for layer 5 can encode completely different information than the prefix for layer 10, letting fine-grained control over how information flows through the network.

In contrast, soft prompts in prompt tuning appear only as additional tokens in the input sequence. They do not directly modify any attention matrix or feed-forward computation. Instead, the model's own attention mechanism determines how much to attend to these soft prompts versus the actual input tokens at each layer. This is a fundamentally more indirect form of intervention: prompt tuning relies on the model learning to use soft prompts through its natural attention patterns rather than injecting information directly into the attention computation. The soft prompts must earn their influence by being useful for the task, as determined by the attention weights the model naturally computes.

This architectural difference has a deeper consequence: because soft prompts only appear in the input layer, they must encode task information in a form that the model can correctly interpret and propagate through many layers of transformation. Each transformer layer applies attention and feed-forward operations that progressively transform the representations. After 12 or 24 layers, the representation of a soft prompt token may look quite different from its original embedding, as the model mixes and transforms information across positions. Smaller models with fewer parameters per layer have limited capacity to preserve task-needed information through this cascade of transformations, which is why they struggle more with prompt tuning than with prefix tuning.

This architectural difference explains why prompt tuning requires larger model scale to match prefix tuning's performance. Smaller models lack the representational capacity to propagate task-needed information from input soft prompts through many layers of transformation without the information being diluted. Larger models, with richer internal representations and more expressive attention patterns per layer, can use input-level modifications more effectively by learning attention patterns that extract and preserve the soft prompt signal across the full depth of the network.