Jailbreaking › Why Safety Training Is Not a Hard Constraint

Before examining attack techniques, it is worth understanding why safety training is vulnerable at all. The answer lies in how alignment is implemented.

RLHF and constitutional AI methods fine-tune models to produce preferred outputs given typical inputs. They don't modify the model's core capabilities or install hard logical constraints. The result is a probability distribution over tokens that, for most inputs, strongly favors safe responses over harmful ones. But "strongly favors" is not the same as "always produces." The model still assigns some probability to harmful continuations; safety training just makes that probability very small for typical inputs.

Think of safety training as reshaping the loss landscape rather than erasing capabilities. Before safety training, the model's weights encode a rich representation of the world, including information about dangerous topics, because that information was present in the pretraining corpus. Safety fine-tuning steers the model away from expressing that information when asked harmful questions, but it does not delete the underlying representations. The weights that encode knowledge of dangerous synthesis routes still exist; the safety gradient has only suppressed the pathways that activate them in response to harmful queries. This is why jailbreaks work: they find alternative activation pathways that the safety gradient did not suppress.

Jailbreaks work by finding inputs for which the probability of harmful completions is not small. These might be inputs that:

• Look syntactically different from training examples (unusual phrasing, roleplay framing)
• Activate the model's helpfulness training more strongly than its refusal training (context manipulation)
• Appear in regions of input space where the safety classifier is poorly calibrated (distribution shift)
• Use the model's own reasoning capabilities against it (logical manipulation)
• Find literally adversarial points in embedding space where harmful outputs have high probability (gradient-based attacks)

The fundamental challenge is that safety training is done on a finite set of examples, but the input space of a language model is essentially infinite. No training set can cover every possible way to phrase a harmful request, so generalization gaps always exist.

Interpretability research offers a complementary view of why jailbreaks succeed. Studies using activation patching and representation probing show that safety-related behaviors are encoded in specific directions in the model's residual stream. When a model processes a harmful request and decides to refuse, identifiable activation patterns in the middle-to-late layers drive the refusal.

This representation structure has two implications. First, jailbreaks that move the model's internal state away from the "refusal direction" should work regardless of the surface-level input manipulation technique used. Second, if you can directly intervene on activations (as in activation steering), you can jailbreak a model without any clever prompting at all: simply subtract the refusal direction from the residual stream during inference.

Representation Engineering (RepE), introduced by Zou et al. (2023b), demonstrated this explicitly. By identifying the "safety" direction in the model's residual stream using linear probes trained on contrasting examples, researchers could directly steer model behavior by adding or subtracting that direction at inference time. This is not a practical attack vector in deployed systems (which don't expose their residual streams), but it confirms that safety training creates identifiable geometric structure in representation space, and that structure can be exploited.