Jailbreaking › Adversarial Suffixes: Gradient-Based Attacks

The manual jailbreak techniques above rely on human creativity and trial-and-error. A more systematic approach uses gradient information to automatically find inputs that cause the model to produce target outputs. This technique, known as Greedy Coordinate Gradient (GCG), was introduced by Zou et al. (2023a) in a paper titled "Universal and Transferable Adversarial Attacks on Aligned Language Models."

The core insight is that jailbreaking can be formulated as an optimization problem. Given a harmful request $x_{\text{harmful}}$, we want to find a suffix $s$ such that when we append $s$ to the request, the model produces a target output (typically "Sure, here is how to...").

Formally, let $p_\theta(y \mid x)$ denote the probability the model assigns to output $y$ given input $x$, where $\theta$ represents the model's fixed parameters (frozen after training). We want to find the suffix $s^*$ that maximizes the probability of a target affirmative response:

$$s^* = \arg\max_{s} \; p_\theta(y_{\text{target}} \mid x_{\text{harmful}} \oplus s)$$

where:

• $s^*$: the optimal adversarial suffix, a sequence of $m$ tokens appended to the harmful request
• $x_{\text{harmful}}$: the original harmful request tokens (e.g., "How do I build a weapon?")
• $\oplus$: token sequence concatenation operator
• $y_{\text{target}}$: the affirmative target prefix we want the model to produce (e.g., "Sure, here is how to...")
• $p_\theta(\cdot \mid \cdot)$: the model's conditional probability distribution over token sequences

Maximizing $p_\theta(y_{\text{target}} \mid x_{\text{harmful}} \oplus s)$ is equivalent to minimizing its negative log-likelihood, which gives us a loss function we can optimize.

The problem is that $s$ is a discrete sequence of tokens, and gradient descent requires continuous parameters. GCG handles this through a token-level coordinate descent:

1. Initialize $s$ as a sequence of random tokens
2. For each position $i$ in $s$: a. Compute the gradient of the loss with respect to the one-hot token representation at position $i$ b. Use the gradient to identify the top-$k$ candidate replacement tokens at position $i$
3. Randomly sample from the candidate replacements across positions
4. Accept a replacement if it decreases the loss
5. Repeat until convergence or a maximum number of steps

The resulting suffix looks like random characters to a human reader, something like:

But this seemingly nonsensical text, when appended to a harmful request, causes the model to begin its response with an affirmative, effectively bypassing safety training.

The mechanism is subtle and reveals something important about how safety training operates. The model's refusal behavior is implemented as a learned mapping from input to output token probabilities. That mapping has a complex, high-dimensional structure. GCG finds a path through that structure to a region where harmful completions have high probability.

Intuitively, the suffix doesn't need to be semantically coherent. It needs to push the model's internal state to a point in embedding space where the next-token distribution favors affirmative responses. Since the suffix is applied after the harmful request, it effectively overrides whatever safety response the request alone would trigger. The model's residual stream at the final position, which determines the next-token distribution, has been shifted by the nonsensical suffix tokens into a region associated with affirmative generation.

The mathematical formulation of GCG can be written more precisely. Let $x = [x_1, x_2, ..., x_n, s_1, s_2, ..., s_m]$ be the full input token sequence: the original request of $n$ tokens followed by the suffix of $m$ tokens. The GCG loss function is the negative log-likelihood of producing the target affirmative prefix:

$$\mathcal{L}(s) = -\log p_\theta(y_{1:T} \mid x) = -\sum_{t=1}^{T} \log p_\theta(y_t \mid x, y_{1:t-1})$$

where:

• $\mathcal{L}(s)$: the loss to minimize; high loss means the model is unlikely to produce the target prefix
• $y_{1:T}$: the target affirmative prefix of $T$ tokens (e.g., "Sure, I'd be happy to help. Here is how to...")
• $y_t$: the $t$-th token of the target prefix
• $y_{1:t-1}$: all target tokens generated before step $t$ (teacher-forced during optimization)
• The sum decomposes the joint probability as a product of conditional probabilities via the chain rule

To minimize $\mathcal{L}(s)$ over the discrete token sequence $s$, GCG uses a linearized approximation. For position $i$ in the suffix, represent the current token as a one-hot vector $e_{s_i} \in \{0,1\}^{|V|}$ where $|V|$ is the vocabulary size. The gradient with respect to this representation is:

$$\nabla_{e_{s_i}} \mathcal{L}(s) = \frac{\partial \mathcal{L}}{\partial e_{s_i}}$$

where:

• $\nabla_{e_{s_i}} \mathcal{L}(s)$: a vector of length $|V|$, one entry per vocabulary token
• Entry $v$ of this gradient estimates how much the loss would decrease if token at position $i$ were replaced by vocabulary token $v$
• Tokens with the most negative gradient entries are the best candidates for replacement

This gradient indicates how the loss changes if token $s_i$ were replaced by each vocabulary token. The top-$k$ tokens with the steepest negative gradient (largest predicted loss decrease) become candidates for replacement at that position.

The most alarming finding from the GCG paper was that the suffixes generalize. A single suffix could be trained to work across many different harmful behaviors (universality), and suffixes trained on open-weight models transferred to commercial black-box models like GPT-4 and Claude (transferability).

Universal suffixes are found by optimizing over a batch of harmful behaviors simultaneously. Instead of minimizing the loss for a single target, the optimization minimizes the average loss across all behaviors:

$$\mathcal{L}_{\text{universal}}(s) = \frac{1}{B} \sum_{b=1}^{B} \mathcal{L}_b(s)$$

where:

• $\mathcal{L}_{\text{universal}}(s)$: the universal loss averaged over all $B$ behaviors
• $B$: the number of distinct harmful behaviors in the training batch (e.g., 25 different harmful request types)
• $\mathcal{L}_b(s)$: the GCG loss for behavior $b$, measuring how well the suffix causes the model to comply with harmful request $b$

A suffix that minimizes this average simultaneously jailbreaks all $B$ behaviors. Finding such a suffix is harder than optimizing for a single behavior because the suffix must work across diverse request types, but this universality makes the attack far more dangerous in practice.

Transferability is less well understood theoretically. The prevailing hypothesis is that aligned models share similar geometric structure in how they implement safety behaviors, so adversarial examples in one model's representation space are also adversarial in another's. This is analogous to adversarial examples in image classification, where perturbations found on one architecture transfer to others. Because many commercial models are fine-tuned from related base models (often in the LLaMA or GPT family), their safety mechanisms may share structural similarities that explain cross-model transfer. However, transfer rates vary considerably by model, and some models exhibit much better cross-architecture resistance than others.

GCG requires white-box access to the model: you must be able to compute gradients through the model's token embeddings. This limits the attack to open-weight models where the architecture and weights are publicly available. Another practical limitation is computational cost: finding a successful adversarial suffix typically requires thousands of forward and backward passes. This is feasible on consumer hardware for small models but requires significant GPU time for models with tens of billions of parameters.

A third limitation is that GCG suffixes are fragile. A successful suffix may fail if the model is updated, if the system prompt changes, or even if small benign modifications are made to the input. This brittleness is a consequence of optimizing in a high-dimensional discrete space: the adversarial point found may lie in a narrow valley that is easily escaped by small input perturbations. This brittleness also suggests a defensive strategy: randomized input transformations that preserve semantics but perturb token sequences can break many GCG suffixes.

A fourth limitation is that GCG optimizes for a single, fixed target prefix. The optimization drives the model to begin its response with "Sure, here is how to..." but does not guarantee that the rest of the response will be useful to the attacker. In practice, even a successful jailbreak (measured as producing the target prefix) may be followed by a vague, unhelpful, or hallucinated continuation. The attacker has produced the target prefix without producing useful content. Stronger attacks combine GCG with PAIR-style iterative refinement to ensure the continuation is also useful.

A fifth limitation is evasion detection. Real-world deployed models are often wrapped with monitoring layers that detect unusual inputs. A suffix consisting of 20 tokens of apparent gibberish is trivially identified as suspicious, even without understanding what it does. This is a practical deployment barrier for GCG that pure optimization metrics don't capture. Attackers must balance the optimization pressure for a lower safety loss against the steganographic pressure for a lower suspicion score, and these objectives often conflict.