Activation Steering: Steering Vectors and Representation Engineering › Applying Steering Vectors at Inference

Once you have a steering vector $\mathbf{v}$, applying it is straightforward. During a forward pass, after computing the residual stream activations at layer $\ell$, you add a scaled version of the vector before passing activations to the next layer:

where:

• $\mathbf{h}_\ell$: the residual stream activations at layer $\ell$ (shape: tokens $\times$ hidden dimension)
• $\alpha$: the steering coefficient, a scalar multiplier controlling intervention strength
• $\mathbf{v}$: the steering vector, a vector in the hidden dimension

Adding $\alpha \mathbf{v}$ with $\alpha > 0$ pushes activations toward the positive concept. Subtracting (using $\alpha < 0$) pushes away from the concept.

The choice of $\alpha$ is empirical and context-dependent. Small values produce subtle behavioral shifts; large values produce strong shifts but can degrade output quality or cause incoherence. A typical range for well-calibrated steering vectors is $\alpha \in [5, 30]$ when the vector has unit norm, but the right value depends on the model size, the concept, and the desired strength of intervention. This is one reason to normalize the steering vector to unit length before applying it: it makes the coefficient interpretable as a distance in activation space rather than an arbitrary scaling that depends on how the vector was constructed.

The injection can be applied at different token positions with different effects. Applying it at every position during generation creates a persistent context: every token the model generates is conditioned on the steered representation throughout. Applying it only at prompt tokens sets an initial direction that influences how the model interprets the prompt. Applying it only at generated tokens modifies generation without changing prompt processing. The most consistent approach, and the one that produces the strongest effects, is constant injection at all positions during generation.

In practice, the injection is implemented via a forward hook: a callback function registered on a specific layer that intercepts the layer's output, adds the scaled steering vector, and returns the modified output. This requires no modification to the model weights and can be added and removed dynamically. The PyTorch hook system makes this straightforward, as we will see in the implementation section.

A natural extension is to inject the steering vector at multiple layers simultaneously. If the concept is encoded across several layers, injecting at all of them may produce a stronger, more consistent effect than injection at a single layer.

However, multi-layer injection can also amplify side effects and cause interactions between the injections at different layers. If the injection at layer $\ell$ shifts activations in a way that changes what layer $\ell+1$ computes, and then another injection at $\ell+1$ adds on top of that, the combined effect may not be a simple superposition of the individual effects. Empirically, single-layer injection at a carefully chosen middle layer often works as well as multi-layer injection with less risk of degradation.