Prompt Tuning › Worked Example: Tracing a Soft Prompt Through Attention

To build concrete intuition for how soft prompts influence model behavior, let's trace through what happens to a soft prompt token when a simplified 2-layer transformer processes it.

Suppose we have a binary sentiment classifier and we have learned a soft prompt with $p = 2$ tokens. After initialization and some training, our prompt matrix is:

$$\mathbf{P} = \begin{bmatrix} 0.8 & -0.3 & 0.5 & \cdots \\ -0.2 & 0.7 & -0.1 & \cdots \end{bmatrix} \in \mathbb{R}^{2 \times d}$$

Our input is "Great movie" with token embeddings $\mathbf{X} \in \mathbb{R}^{2 \times d}$.

Step 1: Form the augmented input. We concatenate to get $\mathbf{X}' = [\mathbf{P}; \mathbf{X}] \in \mathbb{R}^{4 \times d}$. The sequence is now: [prompt_token_0, prompt_token_1, "Great", "movie"].

Step 2: Layer 1 attention. Each of the 4 positions computes queries, keys, and values. When the "Great" token computes its attention weights over all 4 positions, it attends to the two soft prompt tokens as well. If the task is sentiment classification, the soft prompt tokens may have learned to give representations that, when attended to by content tokens, reinforce positive or negative signal. The attention weight from "Great" to prompt_token_0 might be high if prompt_token_0 has learned a key vector aligned with the query pattern of positive sentiment words.

Step 3: Layer 1 output. The attended representations mix information from all four positions. The output at the "Great" position now contains a blend of the token's own meaning and contributions from the soft prompts. The soft prompts have influenced this representation without any special awareness that they are "prompts" rather than regular tokens.

Step 4: Layer 2 attention. The process repeats with the updated representations. By now, the representation at each position has been shaped by the soft prompt influence from layer 1. The soft prompt tokens themselves also have updated representations that incorporate information from the input tokens, letting them to adapt their influence based on the actual content.

Step 5: Classification. The final representation at the classification token (or the mean-pooled representation) captures both the content of the input and the task framing provided by the soft prompts. Because the soft prompts have been optimized to produce the correct classification signal, the final representation should strongly activate the positive sentiment class output.

The key insight from this trace is that soft prompts do not act as simple instructions prepended to text. They act as learned context that interacts with the input through multiple rounds of attention, progressively shaping representations in a way that the model has learned to use for the target task. This interaction is richer and more dynamic than any discrete text prompt could reach, because the soft prompts can encode patterns in the continuous embedding space that have no discrete token equivalent.