Tokenization Challenges › Adversarial Tokenization

The gap between human-readable text and tokenizer-produced token sequences opens a category of adversarial attacks that exploit this mismatch specifically. Unlike adversarial examples in vision, which require computational search for perturbations that fool classifiers, many tokenization-based attacks are simple: they rely on basic properties of Unicode encoding that are easy to apply once understood. The attacker does not need sophisticated optimization. They need knowledge of how the tokenizer behaves on specific character types.

This makes tokenization-based attacks more accessible and more pervasive than many other attack categories. A malicious user who discovers that inserting a zero-width space between the characters of a prohibited keyword causes that keyword to evade a token-based filter does not need machine learning expertise. They need to know that zero-width spaces exist and that they affect tokenization. This knowledge is increasingly widespread.

Prompt injection attacks sometimes exploit tokenization to hide control instructions from human reviewers while ensuring the model receives and executes those instructions. Consider a scenario where a content review system inspects text before it reaches a language model. If the review system operates on human-readable text and the injection is composed of characters that are invisible to humans but visible to the tokenizer, the injection might pass the human review and reach the model.

By inserting invisible Unicode characters between the characters of a harmful command, an attacker can make the command appear as innocuous text to human reviewers while ensuring the tokenizer still produces tokens that represent the command to the model. If the model has learned to follow instructions embedded in its input, and if the tokenized instruction is recognizable enough for the model to parse, the attack succeeds. The key is that the tokenizer sees the raw character sequence, including invisible characters, while the human reviewer sees only the rendered visual representation.

Conversely, attackers can use homoglyphs: characters that look visually identical to common ASCII letters but are different Unicode code points from a different block. Cyrillic а (U+0430, Cyrillic small letter a) looks indistinguishable from Latin a (U+0061) in most fonts. The string аdmin with a Cyrillic а looks exactly like admin with a Latin a. A filter that checks for the exact ASCII string admin will not match the Cyrillic version. If the model's behavior is triggered by a token that happens to be shared (because BPE sees the byte sequence and the byte values are the same), or if the model has learned to treat visually similar tokens as equivalent, the homoglyph might still achieve the attacker's goal while evading ASCII-based filters.

Some jailbreak attempts exploit the fact that a harmful keyword split across token boundaries might not be recognized by a classifier trained on token sequences. If a safety classifier was trained to detect the token "bomb" but the tokenizer in a particular context splits the word as ["bo", "mb"] or ["b", "omb"], the classifier may not activate. The harmful intent is present in the text, and a human reader would recognize it, but the token sequence that the classifier sees does not match the pattern it was trained to detect.

This attack is more reliable than it might seem, because tokenization is context-sensitive. The same word can tokenize differently depending on what precedes it, what case it appears in, and whether it appears with or without a preceding space. An attacker who probes a system can discover which tokenization context causes a keyword to split favorably, then craft inputs that reliably produce that context.

More sophisticated jailbreak techniques encode instructions in transformed forms that the model can decode but that automated systems may not recognize. Reversing text character by character, encoding with simple substitution ciphers, using Pig Latin, spacing out letters, or representing text in a programming language that evaluates to the harmful string are all techniques that have been demonstrated in practice. The effectiveness of these approaches depends on whether the underlying model has learned to process such transformations (which capable models often have, through training on diverse text) and on whether the safety systems can recognize the encoding.

The key insight is that safety systems that operate at the token level are inherently limited by the tokenizer's behavior. If the attacker can manipulate which tokens are produced from a given piece of text, they can potentially manipulate whether safety classifiers activate. This motivates safety approaches that operate at multiple levels: token-level classification, character-level normalization, and semantic understanding of meaning regardless of surface form.

Defending against adversarial tokenization requires understanding the full attack surface and applying mitigations at multiple points in the pipeline. No single defense is sufficient. The most robust approach layers multiple strategies.

Unicode normalization is the foundational defense. Applying NFKC normalization before tokenization collapses homoglyph variants (Cyrillic and Latin characters that appear identical are both mapped to their canonical forms), resolves combining character alternatives, and eliminates compatibility ambiguities. This substantially reduces the surface area for homoglyph attacks and encoding-based confusion, though it does not eliminate all possibilities because not all homoglyphs are Unicode equivalents.

Invisible character stripping removes zero-width spaces, zero-width non-joiners, soft hyphens, byte-order marks, and bidirectional control characters that can be inserted to manipulate tokenization without affecting visual appearance. This is a straightforward preprocessing step that eliminates a whole class of injection attacks. The characters being stripped serve no legitimate purpose in most NLP applications, so their removal does not damage the content.

Input validation can reject text containing characters outside the expected range for the application's use case. A customer service chatbot that expects English text can reject inputs containing Cyrillic or Arabic characters (unless the application explicitly supports those languages). A code completion tool can validate that inputs contain only characters that appear in source code. These constraints reduce the attack surface by preventing the input of exotic Unicode constructions that only make sense in an adversarial context.

Ensemble detection uses multiple tokenization schemes in parallel. If a safety classifier is trained on one tokenizer's output and an attacker exploits that tokenizer's behavior to produce a non-triggering token sequence, a second classifier trained on a different tokenizer's output might still detect the harmful content. Combining the outputs of classifiers with different tokenization schemes makes it substantially harder for an attacker to find a single input that simultaneously evades all of them, because the input would need to exploit different vulnerabilities in multiple tokenizers at once.