Prompt Tuning › Prompt Initialization Strategies

How you initialize the soft prompt matrix $\mathbf{P}$ directly affects whether training converges quickly, converges to a good solution at all, or produces suboptimal results. This question matters more for prompt tuning than for most neural network components, because the embedding space has a specific structure: the transformer was pretrained to process vectors from the vocabulary embedding distribution, not arbitrary random vectors. Starting the soft prompts far from this distribution may create representations the frozen model cannot interpret during the first steps of training, which can lead to poor gradients and slow or unstable optimization.

Think of initialization as choosing the starting point in a large parameter space. If you start in a sensible region of the embedding space, gradient descent can make real steps from the beginning. If you start far outside the region the model has learned to process, you may spend many optimization steps just getting to the border of real space before you can start making productive task-needed updates. For a model whose vocabulary embeddings span a particular manifold in $\mathbb{R}^d$, soft prompts initialized near that manifold receive richer gradient information immediately, because the frozen model's attention heads can compute real similarity scores and produce informative outputs right from the first forward pass.

The simplest approach initializes each prompt embedding from a random distribution:

$$\mathbf{P}_{ij} \sim \mathcal{U}(-a, a) \quad \text{or} \quad \mathbf{P}_{ij} \sim \mathcal{N}(0, \sigma^2)$$

where:

• $\mathbf{P}_{ij}$: the element at row $i$, column $j$ of the prompt matrix (the $j$-th dimension of the $i$-th virtual token)
• $\mathcal{U}(-a, a)$: a uniform distribution bounded symmetrically by $a$, creating values in $[-a, a]$
• $\mathcal{N}(0, \sigma^2)$: a normal distribution with mean zero and variance $\sigma^2$

The hyperparameters $a$ or $\sigma$ should be chosen to match the scale of the model's existing token embeddings. A common choice is $\sigma = 0.02$, which roughly matches the initialization scale of transformer weight matrices. If these values are set too large, the initial soft prompts will lie far outside the distribution of vocabulary embeddings, creating a difficult starting point. If set too small, the soft prompts may start nearly at zero and lack enough spread to create diverse virtual tokens.

Random initialization gives no task-related inductive bias. The optimization process must discover useful prompt embeddings entirely from the training signal, starting from arbitrary points in embedding space. This approach can work reasonably well for large models with sufficient capacity to recover from poor starting points, but it consistently underperforms better initialization strategies at smaller scales and with limited training data.

A more informed strategy initializes prompt embeddings using existing token embeddings from the model's vocabulary. The intuition is direct: the frozen model has been trained to process vocabulary embeddings effectively. By starting soft prompts in the same region of embedding space as vocabulary tokens, we give optimization a head start from a point the model already knows how to interpret.

For each prompt position $i$, you sample a random token $t_i$ from the vocabulary and use its corresponding embedding:

$$\mathbf{P}_i = \mathbf{E}_{t_i}$$

where:

• $\mathbf{P}_i \in \mathbb{R}^d$: the initialization vector for the $i$-th prompt token (a row of $\mathbf{P}$)
• $\mathbf{E} \in \mathbb{R}^{|V| \times d}$: the model's frozen embedding matrix, with $|V|$ vocabulary tokens
• $t_i \in \{0, 1, \ldots, |V|-1\}$: the index of the randomly sampled vocabulary token for position $i$

This initialization ensures soft prompts start in embedding space that the model already processes reliably. The attention mechanisms are designed to handle vectors from this distribution, so starting nearby accelerates optimization. During training, the prompts can move away from their initial positions to better serve the task, but they begin in territory the model is already equipped to interpret. The particular tokens chosen for initialization matter less than the fact that they come from the correct distribution.

Why does this formula make sense? Notice that vocabulary embeddings live on a learned manifold shaped by the model's pretraining. Starting on this manifold means the frozen model's attention heads can compute real similarity scores between the initial soft prompts and the real input tokens immediately. This gives useful gradient signal from the very first training step rather than requiring many steps of "warm-up" before the prompts enter the real region of embedding space.

For classification tasks, we can make vocabulary initialization more targeted by selecting tokens related to task or class labels. For sentiment classification, instead of randomly sampling vocabulary tokens, we might initialize using the embeddings of tokens like "positive", "negative", "sentiment", "opinion", "good", "bad":

$$\mathbf{P}_i = \mathbf{E}_{\text{class\_token}_i}$$

where:

• $\mathbf{P}_i$: the initialization vector for the $i$-th prompt token
• $\mathbf{E}$: the pretrained embedding matrix
• $\text{class\_token}_i$: the token index corresponding to a class label or task-needed word for position $i$

This initialization gives a strong semantic inductive bias: the soft prompts start already encoding task-needed semantics. Gradient descent then refines these semantically-initialized vectors to capture the precise patterns needed for the classification boundary, rather than building semantic content from scratch.

In effect, class-label initialization is like telling the model: "start by thinking about these concepts" and letting the optimization figure out exactly how to combine and transform them. For a three-way sentiment classifier (positive, neutral, negative), you might use the embeddings of "positive", "neutral", and "negative" for three of your prompt tokens, then fill remaining positions with vocabulary tokens related to evaluation or opinion. Training then sharpens these initial representations into the precise combination needed.

For tasks with clear lexical class representations, this initialization can materially reduce the number of training steps needed to reach good performance. It in effect gives a warm start that encodes the classification ontology directly in the initial prompt, giving the optimizer a strong prior about the structure of the task.

The original prompt tuning paper by Lester et al. found that initialization strategy matters most at smaller model scales, a result that confirms the theoretical reasoning above. For T5-Small (60 million parameters), class-label initialization materially outperformed random initialization by several percentage points on benchmark tasks. The gap was large enough to change conclusions about whether prompt tuning was practically viable for smaller models at all. As model scale increased through T5-Base and T5-Large, the gap narrowed progressively. At T5-XXL with 11 billion parameters, all three strategies converged to similar final performance, with differences shrinking to within measurement noise.

This convergence at scale shows an important property of large models: they have enough representational capacity and gradient information to correct for poor starting points during optimization. A large model can, in effect, overcome a bad initialization by learning to work around the poorly-initialized soft prompt tokens and instead encode useful information in other directions. Smaller models lack this overcorrecting capacity, so initialization quality directly determines final performance.

In practice, vocabulary or class-label initialization adds minimal implementation complexity: a few lines of code to sample from the embedding matrix or look up specific token embeddings. This small investment gives a real safety margin against poor convergence, particularly when working with models below the billion-parameter scale. The recommendation is to use vocabulary initialization as the default and class-label initialization when you have a classification task with clear lexical labels.