Tokenization Challenges › Code Implementation
Language AI HandbookAll chapters Tokenization Challenges › Code ImplementationOne section · Sections in this chapter9 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's work through these tokenization challenges with concrete examples using the HuggingFace tokenizers library. We'll examine how GPT-2's BPE tokenizer handles numbers, code, multilingual text, and emoji, observing each failure mode directly.
# Install tokenizers if not present
# uv pip install transformers tokenizers
from transformers import GPT2Tokenizer
# Load GPT-2 tokenizer (BPE, trained on English text)
gpt2_tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
def show_tokenization(text, tokenizer, label=""):
tokens = tokenizer.tokenize(text)
ids = tokenizer.encode(text, add_special_tokens=False)
n_tokens = len(tokens)
return {
"label": label,
"text": text,
"tokens": tokens,
"ids": ids,
"n_tokens": n_tokens,
}Number Tokenization
Let's see how numbers of different sizes and formats tokenize. We expect to observe single tokens for small, frequent numbers and fragmentation for larger, rarer ones.
number_examples = [
"42",
"100",
"1000",
"12345",
"1234567",
"3.14159",
"1,234,567",
"0.001",
"1e-5",
"2024",
]
number_results = []
for num in number_examples:
result = show_tokenization(num, gpt2_tokenizer, label=num)
number_results.append(result)Number Tokenization with GPT-2 BPE Tokenizer ======================================================= Number # Tokens Tokens ------------------------------------------------------- 42 1 '42' 100 1 '100' 1000 1 '1000' 12345 2 '123' | '45' 1234567 3 '123' | '45' | '67' 3.14159 4 '3' | '.' | '14' | '159' 1,234,567 6 '1' | ',' | '234' | ',' | '5' | '67' 0.001 3 '0' | '.' | '001' 1e-5 4 '1' | 'e' | '-' | '5' 2024 2 '20' | '24'
Notice how small numbers like 42 and 100 get single tokens because they appeared frequently in training text. Larger numbers like 1234567 get fragmented, breaking the numeric structure. The number 1,234,567 (with commas) tokenizes differently than 1234567 even though they represent the same quantity. This inconsistency is exactly the problem: the model receives different token sequences for semantically equivalent numbers, creating extra learning burden for any task involving numeric comparison or reasoning.
# Compare arithmetic expressions to see how fragmentation affects multi-number sequences
arithmetic_examples = [
"1 + 1 = 2",
"123 + 456 = 579",
"9999 + 1 = 10000",
"2 * 8 = 16",
"1000 / 4 = 250",
]
arith_results = []
for expr in arithmetic_examples:
result = show_tokenization(expr, gpt2_tokenizer, label=expr)
arith_results.append(result)Arithmetic Expression Tokenization ====================================================================== Expression # Tokens Tokens ---------------------------------------------------------------------- 1 + 1 = 2 5 '1' | 'Ġ+' | 'Ġ1' | 'Ġ=' | 'Ġ2' 123 + 456 = 579 7 '123' | 'Ġ+' | 'Ġ4' | '56' | 'Ġ=' | 'Ġ5' | '79' 9999 + 1 = 10000 5 '9999' | 'Ġ+' | 'Ġ1' | 'Ġ=' | 'Ġ10000' 2 * 8 = 16 5 '2' | 'Ġ*' | 'Ġ8' | 'Ġ=' | 'Ġ16' 1000 / 4 = 250 5 '1000' | 'Ġ/' | 'Ġ4' | 'Ġ=' | 'Ġ250'
The expression 9999 + 1 = 10000 reveals the carry propagation problem. The model must process 9999 as either a single token or multiple fragments, then produce 10000 as its output. Because 10000 has a different tokenization than 9999 + 1, the model cannot implement a simple character-level carry algorithm. It must have learned the relationship between the input and output token sequences from examples.
Code Tokenization
Let's examine how Python code fragments when processed by a tokenizer trained on English prose:
code_examples = [
"def calculate_moving_average(data, window_size):",
"getUserPreferences(user_id, default=None)",
"transformer_attention_weights = torch.softmax(scores, dim=-1)",
"if x > 0 and y < threshold:",
" return {'result': value, 'status': 'success'}",
"for i in range(len(tokens)):",
]
code_results = []
for code in code_examples:
result = show_tokenization(code, gpt2_tokenizer, label=code[:40])
code_results.append(result)Code Tokenization with GPT-2 BPE Tokenizer
=================================================================
Code: def calculate_moving_average(data, window_size):
Tokens (13): ['def', 'Ġcalculate', '_', 'moving', '_', 'average', '(', 'data', ',', 'Ġwindow', '_', 'size', '):']
Code: getUserPreferences(user_id, default=None)
Tokens (13): ['get', 'User', 'Pref', 'erences', '(', 'user', '_', 'id', ',', 'Ġdefault', '=', 'None', ')']
Code: transformer_attention_weights = torch.softmax(scores, dim=-1)
Tokens (20): ['trans', 'former', '_', 'att', 'ention', '_', 'weights', 'Ġ=', 'Ġtorch', '.', 'soft', 'max', '(', 'sc', 'ores', ',', 'Ġdim', '=-', '1', ')']
Code: if x > 0 and y < threshold:
Tokens (9): ['if', 'Ġx', 'Ġ>', 'Ġ0', 'Ġand', 'Ġy', 'Ġ<', 'Ġthreshold', ':']
Code: return {'result': value, 'status': 'success'}
Tokens (17): ['Ġ', 'Ġ', 'Ġ', 'Ġreturn', 'Ġ{', "'", 'result', "':", 'Ġvalue', ',', "Ġ'", 'status', "':", "Ġ'", 'success', "'", '}']
Code: for i in range(len(tokens)):
Tokens (12): ['for', 'Ġi', 'Ġin', 'Ġrange', '(', 'len', '(', 't', 'ok', 'ens', ')', '):']calculate_moving_average gets split at non-semantic boundaries. The tokenizer has no knowledge that shows mark word boundaries in Python identifiers. The camelCase identifier getUserPreferences also fragments at points that do not align with the conceptual components get, User, and Preferences. Each fragment must be processed as a separate token by the model. This consumes context budget and potentially obscuring the relationship between semantically related identifiers.
Fertility Comparison Across Languages
Let's measure fertility across different languages using semantically equivalent content. We use "The quick brown fox jumps over the lazy dog" translated into multiple languages, which gives us comparable semantic content for the comparison.
# "The quick brown fox jumps over the lazy dog" in multiple languages
multilingual_sentences = {
"English": "The quick brown fox jumps over the lazy dog",
"German": "Der schnelle braune Fuchs springt über den faulen Hund",
"French": "Le rapide renard brun saute par-dessus le chien paresseux",
"Spanish": "El rápido zorro marrón salta sobre el perro perezoso",
"Finnish": "Nopea ruskea kettu hyppää laiskan koiran yli",
"Arabic": "الثعلب البني السريع يقفز فوق الكلب الكسول",
"Japanese": "素早い茶色のキツネは怠け者の犬を飛び越える",
"Chinese": "敏捷的棕色狐狸跳过懒惰的狗",
}
# Count words manually (whitespace-split for rough approximation)
def word_count(text):
return len(text.split())
fertility_results = []
for lang, sentence in multilingual_sentences.items():
tokens = gpt2_tokenizer.tokenize(sentence)
n_tokens = len(tokens)
n_chars = len(sentence)
# For non-spaced languages, use character count as denominator for per-char fertility
n_words = (
word_count(sentence)
if lang not in ["Japanese", "Chinese", "Arabic"]
else max(1, n_chars // 3)
)
fertility = n_tokens / n_words
fertility_results.append(
{
"language": lang,
"n_tokens": n_tokens,
"n_words": n_words,
"fertility": fertility,
}
)GPT-2 BPE Tokenizer Fertility by Language ======================================================= Language Tokens Words/Units Fertility ------------------------------------------------------- English 9 9 1.00 German 19 9 2.11 French 20 9 2.22 Spanish 22 9 2.44 Arabic 39 13 3.00 Finnish 22 7 3.14 Japanese 36 7 5.14 Chinese 29 4 7.25
The fertility disparity is stark. English achieves near-ideal fertility because the tokenizer was trained on English text and has efficient vocabulary coverage for English words. Non-Latin languages require far more tokens to represent equivalent content. This means non-English users effectively have a smaller context window when using English-trained tokenizers. For a model with a 4,096-token budget, the practical difference between English and Arabic fertility translates to a real difference in how much information each user can include in a prompt.
Emoji and Unicode Edge Cases
unicode_examples = [
("Simple ASCII", "Hello world"),
("Accented (NFC)", "café résumé naïve"),
("Emoji simple", "I love Python 🐍"),
("Emoji complex family", "👨👩👧👦 went to the park"),
("Flag emoji", "The flag is 🇺🇸"),
("Skin tone", "👍🏾 good work"),
("Math symbols", "∀x ∈ ℝ: x² ≥ 0"),
("Mixed scripts", "Hello мир 你好"),
]
unicode_results = []
for label, text in unicode_examples:
result = show_tokenization(text, gpt2_tokenizer, label=label)
unicode_results.append(result)Unicode and Emoji Tokenization ====================================================================== Simple ASCII: 'Hello world' Tokens (2): ['Hello', 'Ġworld'] Accented (NFC): 'café résumé naïve' Tokens (7): ['c', 'af', 'é', 'Ġré', 'sum', 'é', 'Ġnaïve'] Emoji simple: 'I love Python 🐍' Tokens (6): ['I', 'Ġlove', 'ĠPython', 'ĠðŁ', 'IJ', 'į'] Emoji complex family: '👨\u200d👩\u200d👧\u200d👦 went to the park' Tokens (18): ['ðŁij', '¨', 'âĢ', 'į', 'ðŁij', '©', 'âĢ', 'į', 'ðŁij', '§', 'âĢ', 'į', 'ðŁij', '¦', 'Ġwent', 'Ġto', 'Ġthe', 'Ġpark'] Flag emoji: 'The flag is 🇺🇸' Tokens (9): ['The', 'Ġflag', 'Ġis', 'ĠðŁ', 'ĩ', 'º', 'ðŁ', 'ĩ', '¸'] Skin tone: '👍🏾 good work' Tokens (7): ['ðŁij', 'į', 'ðŁ', 'ı', '¾', 'Ġgood', 'Ġwork'] Math symbols: '∀x ∈ ℝ: x² ≥ 0' Tokens (13): ['âĪ', 'Ģ', 'x', 'ĠâĪ', 'Ī', 'Ġâ', 'Ħ', 'Ŀ', ':', 'Ġx', '²', 'Ġâī¥', 'Ġ0'] Mixed scripts: 'Hello мир 你好' Tokens (10): ['Hello', 'ĠÐ', '¼', 'и', 'ÑĢ', 'Ġ', 'ä½', 'ł', 'å¥', '½']
The emoji examples reveal how dramatically token counts can vary. The simple snake emoji 🐍 requires multiple byte-level tokens because it is a non-ASCII code point encoded as four bytes in UTF-8. The family emoji 👨👩👧👦, which looks like a single character in any emoji-aware display, requires far more tokens because it is a sequence of multiple code points joined by zero-width joiners. From the tokenizer's perspective, this "single character" is a complex byte sequence that may expand into ten or more tokens. Math symbols and mixed-script text also reveal the byte-level fallback mechanism that handles characters without dedicated vocabulary entries.
Adversarial Tokenization Examples
Let's demonstrate how invisible Unicode characters and homoglyphs can produce different tokenizations from visually identical text:
# Demonstrate how homoglyphs can change tokenization
# Latin 'a' vs Cyrillic 'а' (they look identical in most fonts)
latin_admin = "admin"
# Cyrillic а (U+0430) looks identical to Latin a
cyrillic_admin = "\u0430dmin" # First char is Cyrillic
examples_adv = [
("Latin 'admin'", latin_admin),
("Cyrillic 'аdmin'", cyrillic_admin),
("Normal word", "running"),
("Inserted zero-width space", "run\u200bning"), # zero-width space mid-word
("Soft hyphen", "run\u00adning"), # soft hyphen mid-word
]
adv_results = []
for label, text in examples_adv:
result = show_tokenization(text, gpt2_tokenizer, label=label)
result["bytes"] = text.encode("utf-8").hex()
adv_results.append(result)Adversarial Tokenization Examples ================================================================= Latin 'admin' Text repr: 'admin' Tokens (1): ['admin'] Cyrillic 'аdmin' Text repr: 'аdmin' Tokens (3): ['а', 'd', 'min'] vs. baseline: DIFFERENT tokenization Normal word Text repr: 'running' Tokens (1): ['running'] Inserted zero-width space Text repr: 'run\u200bning' Tokens (3): ['run', 'âĢĭ', 'ning'] vs. baseline: DIFFERENT tokenization Soft hyphen Text repr: 'run\xadning' Tokens (3): ['run', 'ÂŃ', 'ning'] vs. baseline: DIFFERENT tokenization
The homoglyph example demonstrates that visually identical strings can produce different token sequences. The zero-width space example shows how a completely invisible character can split what would otherwise be a single token into two. These differences are exploitable: a filter that operates on token sequences can be evaded by inserting the right invisible character to cause the target word to split at a boundary the filter does not check.
Visualizing Tokenization Differences
The bar chart reveals a clear pattern: numbers that appeared frequently in the training corpus receive single tokens. Larger or less common numbers fragment into multiple tokens, disrupting the positional structure needed for arithmetic. Notice that the fragmentation is not monotone: 999 might receive more tokens than 1000 if 1000 appeared more frequently in training data as a round number. This frequency-based inconsistency is a fundamental property of the BPE algorithm when applied to numeric text.
Key Parameters for Tokenizer Configuration
Working with tokenizers for robustness and quality involves these important considerations:
add_prefix_space: Many tokenizers treat the first token of a sequence differently (without a leading space). Settingadd_prefix_space=Trueensures consistent tokenization regardless of position in a longer text.truncationandmax_length: Controls how long inputs are handled. High-fertility languages will hit length limits faster than English for the same semantic content.padding: Required for batch processing but consumes token budget. The padding length is determined by the longest sequence in the batch, which for high-fertility languages will be longer.- Normalization settings: In SentencePiece and HuggingFace tokenizers, normalization rules (NFKC, NFC) affect how Unicode is handled before tokenization. Enabling NFKC normalization is the most important defense against Unicode-based attacks.