Activation Steering: Steering Vectors and Representation Engineering › Activation Addition

The term "activation addition" was introduced in a 2023 paper by Turner, Thiergart, Udell, Mini, and MacDiarmid, who demonstrated the technique on GPT-2-XL. Their experiments provide a clean illustration of what steering can and cannot do.

The setup was simple: construct a steering vector by subtracting activations for a "negative" prompt from activations for a "positive" prompt, then add a scaled version of this vector to all forward pass positions at a specific layer. No training, no weight updates, no optimizer. Just arithmetic on activations.

One of their most striking demonstrations involved constructing a "banana" vector by taking the difference between "I talk about bananas all the time." and "I talk about oranges all the time." When this vector was injected with a high coefficient during generation, the model began inserting banana-related content into otherwise unrelated outputs. The model would discuss history but somehow link it to bananas, or write code but reference bananas unprompted. The change extended beyond the word "banana"; the model's conceptual associations shifted, with its reasoning becoming banana-adjacent in context-appropriate ways.

The banana demonstration is deliberately whimsical to make the core point vivid: the vector really does encode something conceptual, not just lexical. If it were only encoding the word "banana," you would see the word inserted at random positions. Instead, the model integrates the banana concept into its outputs in grammatically appropriate ways, treating it as a persistent conceptual context.

More practically, Turner et al. constructed vectors for concepts like "wedding," "anger," "fearful," and "optimistic" and showed that injecting these vectors reliably shifted the model's outputs toward the relevant concept while maintaining grammatical coherence and largely preserving responses to the specific question asked.

The experiments also revealed that the technique has failure modes. At high steering coefficients, outputs can become incoherent or the model can begin to "rant" repetitively. There is a sweet spot between no effect and degraded output, and finding it requires empirical calibration per concept and per model. The sweet spot also shifts with model size: larger models tend to be more robust to high steering coefficients before their outputs degrade, suggesting that bigger models have more capacity to absorb the perturbation without losing coherence.

Activation addition and prompt engineering are both ways to influence model behavior without changing weights, but they operate through completely different mechanisms. A prompt like "Be confident and assertive in your response" works by adding text to the context window that the model attends to when generating. It relies on the model having learned to follow such instructions from its training distribution.

Activation steering bypasses the attention mechanism entirely. It modifies the representation directly, without the model "reading" a steering instruction. This has several implications. Steering is more robust to prompts that would override or dilute the steering instruction: adding a steering vector cannot be "ignored" by the model the way a prompt instruction can. Steering also does not consume context window tokens, making it free in terms of the prompt budget. On the other hand, steering does not benefit from the model's capacity to reason about and interpret complex instructions; it is a blunt geometric intervention.