Tokenization Challenges: Numbers, Code, Multilingual

Subword tokenization algorithms like BPE, WordPiece, and Unigram LM solve the vocabulary explosion problem elegantly. They compress a seemingly infinite space of words into a manageable vocabulary by splitting rare or unseen words into smaller, reusable pieces. The vocabulary might contain thirty thousand entries, but it can represent practically any text by composing those entries. That compression is powerful, and it underlies much of the success of modern language models. But compression always involves a tradeoff, and in tokenization the tradeoff is: the tokenizer does not understand the text it is splitting. It sees a raw character stream and applies its learned merge rules or probability model mechanically, without regard for whether the resulting segments correspond to meaningful units in the input.

The results of that mechanical process are often surprising, occasionally frustrating, and sometimes harmful to model performance. A number gets split in ways that destroy the positional significance of its digits. A function name in a program gets fragmented at a boundary that has no relationship to the semantic components of the name. A sentence in Arabic requires three times as many tokens as its English translation, which means an Arabic user gets a third of the effective context window. An emoji that looks like a single character expands into dozens of tokens because its Unicode encoding is deeply complex. And an attacker who understands tokenization can craft text that evades detection, hides instructions, or manipulates a model's behavior in ways that are invisible to human reviewers.

These failure modes are not exotic corner cases. Every production deployment of a language model eventually encounters them. A financial system that asks a model to reason about prices will hit number tokenization problems. A coding assistant will encounter identifier fragmentation. A product launched in multiple languages will reveal the fertility disparity immediately. A content moderation system will be probed by adversarial inputs that exploit Unicode confusion. Understanding these failure modes shapes how you preprocess data, how you evaluate models, which tokenizer you choose for a new application, and how you design systems that remain robust when faced with the full diversity of real-world input.

This chapter examines each class of tokenization failure systematically. For each one, we first build the conceptual understanding: what goes wrong and why, at the level of the tokenizer's design. We then look at the practical consequences: how the failure affects model performance and user experience. Finally, we discuss mitigations: what you can do as a practitioner to reduce the impact of the failure. The chapter closes with a code walkthrough that lets you observe these failures directly with a real tokenizer, so you develop an intuition for them that goes beyond descriptions.

Think of a tokenizer as a compression algorithm that was designed and trained on a specific kind of data. Just as a JPEG compressor works well on natural photographs but produces ugly artifacts when applied to screenshots with sharp lines and text, a tokenizer trained on English prose works well on English prose but produces artifacts when applied to numeric data, source code, or non-Latin scripts. The compressor is not broken. It is working exactly as designed. The problem is that its design assumptions no longer hold for the input you are giving it.

Numbers appear in almost every kind of text: dates, quantities, prices, coordinates, identifiers, measurements, and statistical results. Yet they represent one of the most persistent and well-documented failure modes for subword tokenizers. The root cause is a fundamental mismatch between how numbers carry meaning and how tokenizers assign tokens.

Numbers derive meaning from their positional value system. The digit 3 in 300 means three hundred because it sits in the hundreds place. The same digit in 30 means thirty, and in 3 means three. A human reader processes this positional structure automatically. We do not need to learn that 300 is one hundred times 3. The decimal positional system encodes that relationship structurally, and our arithmetic education trains us to exploit that structure. Tokenizers learn from co-occurrence statistics. The token that represents "300" appears in contexts like "300 miles" and "300 soldiers", while the token for "3" appears in different contexts. The tokenizer has no mechanism to encode the fact that 300 = 3 × 100. The model must learn that from examples.

This would be manageable if the tokenizer at least segmented all numbers into their individual digits, preserving the positional structure. But most subword tokenizers do not do this. Instead, they treat digit sequences as character sequences to be merged into tokens based on frequency in training data. The number "1000" might become a single token because it appears frequently in English text. The number "9873" might split as ["98", "73"] because those two-digit sequences happened to be frequent subwords. The number "1234567" might split as ["123", "456", "7"] or some other combination that reflects corpus statistics rather than numeric structure. The segmentation is arbitrary from a mathematical standpoint.

The key insight is this: numeric meaning is compositional in a way that statistical co-occurrence cannot capture. When you see the digits 1, 2, 3, 4 appearing together as 1234, you know their value is 1*1000 + 2*100 + 3*10 + 4*1. That composition rule is exact and universal. But a tokenizer trained on statistical patterns sees "1234" as a string and notices that this string appears in contexts similar to "1233" and "1235". It does not see the additive relationship between digits. So the tokenizer might give "1234" its own token (if it was frequent enough in training data) or split it arbitrarily if it was not.

Large numbers suffer the most from this mismatch. A quantity like 4,792,581 is rare in any training corpus. No long numeric subword token exists for it in the vocabulary. The tokenizer fragments it into whatever shorter digit sequences it has available, potentially producing something like ["4", ",", "79", "2", ",", "5", "81"]. Each of those pieces must be processed as a separate token, and the transformer must attend across all of them to reconstruct the full value. This is not how arithmetic works for humans, and it requires the model to develop implicit multi-token numeric reasoning that natural language training data does not directly teach.

Arithmetic is where this failure shows up most acutely and has been studied most carefully. Consider asking a language model to compute 1234 + 5678. The model receives both operands as fragmented token sequences. It must perform multi-digit addition across token boundaries in its attention layers, carrying information about place values from one token to another. Research has consistently shown that transformers trained on text tokenized with standard subword methods struggle with arithmetic beyond a few digits. The failure is not primarily about the model's reasoning capacity. It is substantially about the tokenization scheme not preserving the structural information that makes digit-by-digit arithmetic learnable.

A concrete illustration: the expression 9999 + 1 = 10000 involves a carry that propagates across all four digits of 9999. A human approaches this by adding digit pairs and propagating carries from right to left. This works because the positional structure of the numbers is explicit. But if the tokenizer represents 9999 as a single token and 10000 as a single different token, the model must learn the relationship between these two token identities directly from examples of the computation. It cannot decompose the problem into digit-level steps. If the tokenizer instead splits 9999 as ["99", "99"], the model has two token boundaries to reason across for the input and must produce 10000 which might itself split as ["100", "00"]. Neither representation supports the step-by-step carry propagation that makes addition tractable.

The same problem extends beyond arithmetic. Comparisons between numbers (is 4792581 larger than 4789000?) require the model to compare corresponding digit positions. Date arithmetic (how many days between two dates?) requires understanding both numeric values and calendar structure. Financial calculations (what is the compound interest on a principal of $10,000?) require exact numeric manipulation. In all these cases, fragmented tokenization makes the task harder than it needs to be.

One approach that improves numeric reasoning is tokenizing numbers digit by digit. If each digit 0 through 9 receives its own dedicated token, then every number gets a predictable, positional tokenization regardless of its magnitude or frequency in the training corpus. The number 12345 always becomes ["1", "2", "3", "4", "5"]. The number 9999999 becomes seven digit tokens. There is no ambiguity, no frequency-dependent fragmentation, and no inconsistency between different numeric representations.

This approach preserves the positional structure that arithmetic requires. When the model sees ["1", "2", "3", "4"] + ["5", "6", "7", "8"], the digit-level tokens align naturally with the column-by-column structure of addition. Models trained with digit-level number tokenization show substantially better performance on arithmetic tasks compared to models trained with standard subword tokenization on the same data, even when controlling for model size and training compute.

The tradeoff is token count. A seven-digit number consumes seven tokens instead of one or two with subword tokenization. For a model with a 4,096-token context window, a document containing many large numbers will require significantly more context budget than the same document would with subword tokenization. For applications where numeric reasoning is central, such as mathematical assistants, financial analysis tools, or scientific computing helpers, this tradeoff is often worthwhile. The reduction in arithmetic error rate justifies the increased context consumption.

Some recent models have adopted character-level or byte-level representations specifically to regularize how numbers are handled. GPT-4 and similar models, which show improved arithmetic capabilities, are believed to use tokenization schemes that more consistently segment numbers. Llama 3's tokenizer was specifically updated to tokenize digits individually. This reflects the research consensus that digit-level tokenization improves numeric reasoning. This is a practical solution to one well-understood tokenization failure mode.

If you are building a system that needs to reason carefully about numbers, the tokenization choice is an important design decision. Several strategies are available, and the right choice depends on the application.

Normalizing numeric formats before tokenization reduces variance without changing the tokenizer. Removing thousands-separating commas, standardizing decimal notation, and converting scientific notation to decimal form ensures that the same numeric value consistently produces the same token sequence. This does not solve the fragmentation problem for large numbers, but it eliminates one source of inconsistency that adds noise to the model's numeric understanding.

Using a tokenizer that segments digits individually eliminates the fundamental fragmentation problem. If you are training a model rather than using an existing one, this is worth considering for numeric-heavy domains. If you are using an existing model, check its tokenizer documentation. Models like Llama 3 with single-digit tokenization are preferable for arithmetic-intensive applications.

For applications requiring precise arithmetic, the most reliable approach is to offload computation to an external tool rather than relying on the model's internal representation. When a language model has access to a Python interpreter or a calculator tool, arithmetic accuracy becomes essentially perfect, independent of tokenization. This architecture separates the language model's natural language understanding capabilities from the symbolic computation capabilities that tokenization fundamentally limits.

Programming languages present a different but equally important class of tokenization challenges. Code has rigid syntactic structure: keywords, operators, identifiers, and literals each have precise meanings that depend on their position and relationships within the source text. The indentation of a Python block is syntax, not style. The camelCase capitalization of a JavaScript function name is a convention that carries semantic information to human readers. The underscore-separated components of a Python identifier like calculate_moving_average reflect the conceptual decomposition of the function's purpose. None of this semantic structure is visible to a subword tokenizer trained on natural language.

When a subword tokenizer encounters Python code, it treats it like any other text. It splits identifiers by learned merge rules, handles indentation and whitespace based on statistical patterns from natural language training data, and produces tokenizations that might seem reasonable character-by-character but routinely violate the semantic structure of the code. The tokenizer has no parse table for Python syntax, no understanding of what an identifier is, and no concept of indentation significance.

Think of it this way: a tokenizer trained on English prose is like a newspaper editor who has been handed a page of chemical formulas. The editor can read each character and might notice that some character sequences appear repeatedly, but the editor has no understanding of the structural conventions that give those sequences meaning. The result is a segmentation that serves corpus statistics rather than chemical notation.

Function and variable names in code are often long, descriptive strings constructed from multiple meaningful components. Names like calculate_moving_average, transformer_attention_weights, getUserPreferences, or validate_email_address encode their meaning through their components. A programmer reading calculate_moving_average immediately understands three things: this function calculates something, the thing it calculates is a moving average, and the underscore-separated parts are the conceptual units of that description. This is the entire point of descriptive naming conventions: identifiers are designed to be read and understood by humans in terms of their parts.

Subword tokenizers trained on natural language do not honor these naming conventions. Identifiers like calculate_moving_average appear rarely enough in training data that no single token exists for the whole name. The tokenizer splits it into subword units based on character frequency, potentially producing something like ["calc", "ulate", "_mov", "ing", "_average"]. The grouping reflects which three- and four-character sequences appeared frequently in the training corpus, not the semantic components calculate, moving, and average. The underscore, which a programmer reads as a word separator, is treated as just another character to be merged according to frequency.

This fragmentation has concrete downstream effects. The model needs to attend across five tokens to reconstruct a single conceptual identifier. If the same identifier appears in multiple places in a program, the model must learn that these five-token sequences always represent the same function. More importantly, the model cannot easily learn relationships between semantically related identifiers: the connection between calculate_moving_average and calculate_weighted_average is obvious to a human who sees both are calculating averages, but the tokenizer might split them at different boundaries, making the shared prefix less salient.

Python, YAML, and Markdown use indentation as syntax. A Python function body at four spaces of indentation and the same function body at eight spaces of indentation have fundamentally different meanings: the second is nested inside an additional block. This is not a stylistic choice but a syntactic one. The parser treats indentation as a structural signal equivalent to curly braces in C or Java. Getting the indentation wrong is a syntax error.

Most natural language tokenizers handle whitespace as word-boundary markers. The SentencePiece ▁ prefix approach identifies word-initial positions by noting where spaces precede tokens, which works well for identifying word boundaries in English prose. But four spaces of Python indentation is not a word boundary. It is a block-level structural signal that carries information about which block a statement belongs to. The tokenizer treats it as leading whitespace and may produce a small number of space tokens that do not reflect the hierarchical block structure.

The practical consequence is that a language model processing Python code must learn indentation significance from contextual patterns rather than from explicit structural encoding. It must discover, from training examples, that lines with deeper indentation are syntactically subordinate to lines with shallower indentation, even though the tokenizer provides no direct structural signal for this. Models do learn this from sufficient training data, but the learning task is harder than it would be if the tokenizer explicitly marked indentation levels.

A related issue arises with tabs versus spaces. Python accepts both, and while the PEP 8 style guide recommends spaces, real codebases mix them. A tab character and four space characters look identical in many editors but are different bytes, producing different tokens. The tokenizer has no way to normalize them to equivalent indentation levels.

Code often contains structural repetition that natural language does not: many parentheses, repeated operators, multiple occurrences of the same variable name in close proximity, extensive list comprehensions, deeply nested dictionary literals. These patterns strain tokenizers in ways that natural language does not. Consider a JSON structure with many repeated keys like "id", "name", "value", or a CSV file with numeric data in many rows. The tokenizer produces tokens for each repeated element independently, consuming context window space for content that has high structural redundancy.

A Python list comprehension like [x * 2 for x in range(100) if x % 2 == 0] contains operators, keywords, identifiers, and numeric literals that each need tokens. The total token count for this single expression might be fifteen to twenty tokens, even though the expression has a concise mathematical meaning that a human reads instantly. The context window is consumed by syntactic scaffolding rather than semantic content.

This problem has motivated the development of code-specific tokenizers. Models like Codex (the foundation of early GitHub Copilot), CodeLlama, and StarCoder were trained with tokenizers that include richer code-relevant vocabulary. Common operators, keywords, identifier components, and short identifiers receive dedicated tokens, reducing fragmentation. The result is that a larger fraction of the context window can contain meaningful code content rather than fragmented subword pieces. Studies comparing standard NLP tokenizers to code-specialized tokenizers on programming tasks consistently find that the code-specialized tokenizer improves performance on longer programs, where context efficiency matters most.

The cumulative effect of identifier fragmentation, whitespace handling, and repeated-pattern inefficiency is a significant reduction in effective code reasoning capacity. A model with a 4,096-token context window and a code-specialized tokenizer can process substantially more lines of Python than the same model would with a general-purpose tokenizer. For tasks like code completion, bug fixing, and code review, context capacity directly determines how much surrounding code the model can consider when making a prediction.

Code reasoning also suffers from the lack of structural encoding. A program has a syntax tree: functions contain statements, statements contain expressions, expressions contain identifiers and operators. This tree structure is necessary for understanding control flow, variable scope, and data dependencies. Tokenizers flatten this tree into a linear sequence, discarding the structural information. Models must re-infer the tree from the linear token sequence, which they learn to do with sufficient training but which could be made more efficient with tokenizers that are aware of syntactic structure.

When a tokenizer is trained primarily on English text, non-English languages receive systematically worse tokenization. This causes concrete and measurable downstream effects on model performance, computational cost, and the practical usability of language models for the billions of people who do not primarily use English.

The fundamental issue is that a tokenizer's vocabulary is a finite resource. Thirty thousand or fifty thousand entries can only cover so much linguistic territory. A tokenizer trained on English text fills its vocabulary with English-relevant subwords: common English word fragments, frequent English words, English morphological patterns. When this tokenizer encounters Finnish, Arabic, Chinese, or any other language, it cannot cover those languages' patterns efficiently. It falls back to shorter, more frequent subword fragments or to individual characters, producing many more tokens to represent the same semantic content.

Think of the vocabulary as a set of compression shortcuts. If you have a shortcut for every common English word and word fragment, you can compress English text efficiently. But if you apply that same shortcut table to French text, many French words will not match any shortcut, and you will need to decompose them into smaller pieces. The same information is present, but it takes more space to encode. In the tokenizer context, more space means more tokens, which means more context window usage and more computation.

The concept of fertility measures how many tokens a given amount of text requires. Formally, the fertility of a tokenizer on a corpus is the average number of tokens produced per word (or per character for scripts without word boundaries). A tokenizer with low fertility for a language is efficient for that language: it can pack more semantic content into a fixed number of tokens. A tokenizer with high fertility for a language is inefficient: it uses many tokens to represent the same content, leaving less room in the context window for other information.

English-trained tokenizers typically achieve fertility close to 1.0 for English: roughly one token per word. For morphologically simple language like English, where words rarely have many different endings, this is achievable because common words fit in single tokens. For morphologically rich languages, where the same root can appear with dozens of different suffixes, fertility rises because the tokenizer cannot cover all the surface forms efficiently.

Consider a simple sentence in several languages. The English version tokenizes into approximately 8 tokens. The Finnish version of the same sentence might require 15 tokens because Finnish has complex morphological suffixes that are rare in an English-trained tokenizer's vocabulary. The Arabic version might require 20 tokens because Arabic uses prefixes and suffixes heavily and its script has no equivalents in English vocabulary entries. The Thai version might require even more tokens because Thai has no word boundaries, forcing the tokenizer to process the text character by character or in very short sequences.

The practical consequence of this disparity is stark. If your model has a context window of 4,096 tokens, an English speaker can fit roughly 3,000 English words into a prompt. An Arabic speaker attempting to communicate the same information in Arabic can fit perhaps 1,500 Arabic words, a fifty percent reduction in effective context. They are paying a computational and informational tax for communicating in their native language rather than English. This is not a design goal. It is an unintended consequence of training tokenizers on imbalanced corpora.

This disparity compounds in three ways:

• Shorter effective context: non-English users cannot fit as much content into a fixed context window, which means less relevant context for each query and reduced performance on tasks that benefit from long context.
• Higher computational cost: more tokens means more attention computation. At inference time, processing an Arabic sentence costs more than processing its English translation of equivalent semantic content, making the model more expensive to operate for non-English users.
• Less training signal: the same training corpus that produces a high-English vocabulary also tends to contain more English text, so non-English languages receive less training data in addition to less efficient tokenization, creating a compounding disadvantage.

Different writing systems pose categorically different tokenization challenges that go beyond simple vocabulary coverage. Latin-script languages share a 26-letter alphabet, so tokenization patterns transfer across them to some degree: subwords that appear in English also appear in French, German, and Spanish, giving those languages better coverage than completely different scripts.

Chinese presents a specific challenge because Chinese writing does not use spaces to mark word boundaries. Chinese words can be one, two, or three characters, and the same sequence of characters can be segmented as words in multiple ways depending on context. Most tokenizers handle Chinese by treating each character as a potential token unit. This produces a workable tokenization, but one that does not respect word-level semantics. Two-character compounds that function as single words in Chinese get split into individual character tokens, potentially obscuring the compound meaning.

Japanese adds another layer of complexity by mixing four different scripts in a single sentence: hiragana (phonetic syllabary for grammatical elements), katakana (phonetic syllabary for foreign words and emphasis), kanji (ideographic characters borrowed from Chinese), and often Latin characters for technical terms or brand names. A single Japanese sentence might contain all four scripts. A tokenizer must handle each script transition gracefully, correctly applying the appropriate tokenization rules for each script. Standard English-trained tokenizers typically handle Latin characters in Japanese text as if they were English words, handle kanji with byte-level fallbacks, and handle hiragana and katakana partially or inconsistently.

Arabic adds right-to-left directionality, complex cursive shaping where the appearance of a letter depends on its position in a word, and an affixation system where prefixes and suffixes fuse morphologically with roots. These properties mean that the surface forms of Arabic words are highly diverse, requiring either a very large Arabic-specific vocabulary or aggressive fragmentation. Byte-level tokenizers that process Arabic character by character or byte by byte produce tokenizations that are functional but highly inefficient.

Byte-level tokenizers offer one universal solution: they represent every possible byte as a token. This ensures that any text in any script can be tokenized regardless of its Unicode representation. But byte-level representations of non-Latin scripts are severely inefficient. A single Chinese character encoded in UTF-8 requires three bytes, so a ten-character Chinese sentence becomes thirty byte tokens. Compared to a native Chinese tokenizer that gives each character its own token, byte-level tokenization triples the context window cost. The universality of byte-level tokenization is real, but the efficiency cost for non-Latin scripts is large.

Real multilingual text frequently mixes languages within a single document or even within a single sentence. This phenomenon, known as code-switching in linguistics, is pervasive in online communication. A Spanish-English bilingual speaker might write "I was like, no puede ser, you know?" inserting a Spanish phrase into an otherwise English sentence. A technical forum in German might include English programming terminology throughout a German discussion. Social media posts in Korean might freely mix Korean and English words.

Subword tokenizers trained on multilingual corpora can handle code-switching by having vocabulary entries for both languages. When the tokenizer encounters a Spanish phrase in an English sentence, it applies Spanish-appropriate tokenization to those words and English-appropriate tokenization to the English words. This works reasonably well for neighboring languages with overlapping scripts and borrowings, less well for languages with completely different scripts where the boundary between scripts must be handled as a context switch.

The deeper challenge of code-switching is semantic. When a model encounters a Spanish phrase embedded in English text, it must understand that the Spanish phrase is semantically equivalent to the English word or phrase that would appear in its place. The tokenizer provides no signal about this semantic equivalence. The model must infer from context that "no puede ser" is functioning as an expression of disbelief, equivalent to English expressions like "no way" or "that can't be." Multilingual models trained on large amounts of code-switched text learn these patterns, but the learning is indirect and requires much more training data than a model that had explicit cross-lingual alignment signals would need.

Multilingual models like mBERT and XLM-R address the code-switching challenge by training on large multilingual corpora and learning shared representations for concepts across languages. Because they see translations and multilingual documents, they learn that "run" and "courir" and "laufen" all activate similar representations. But this learning is primarily through shared contextual patterns, not through the tokenizer's structure. The tokenizer itself does not provide cross-lingual signals. The fundamental fertility disparity between English and morphologically complex languages remains even in the most capable multilingual models.

Modern text is not just letters, digits, and punctuation. It includes emoji, mathematical symbols, currency signs, combining characters, special whitespace, bidirectional control characters, and the full breadth of Unicode's 150,000+ code points. Unicode was designed to provide a universal encoding for every character in every writing system in human use. The breadth of that goal means that the encoding is complex, with many special cases, combining rules, and compatibility considerations. Subword tokenizers trained primarily on English text are poorly equipped for this diversity.

The challenge is not just that exotic characters might not appear in the tokenizer's vocabulary. It is that the same visual text can have multiple different encodings, that some characters are invisible to humans but visible to tokenizers, and that some characters combine into compound units that look like single characters but tokenize as many. These properties create opportunities for both unintentional tokenization failures and deliberate adversarial manipulation.

Emoji present a unique challenge that combines complex visual semantics with non-obvious byte encoding. A simple emoji like 🐍 (snake, often used to represent Python) is a single Unicode code point (U+1F40D) encoded as four bytes in UTF-8. A byte-level tokenizer represents it as four tokens. A character-level tokenizer represents it as one token. A subword tokenizer trained on English text might have the snake emoji in its vocabulary if it appeared enough in training data, or it might represent it as a byte sequence if it did not.

More complex emoji are not single code points at all. The family emoji 👨‍👩‍👧‍👦 is a sequence of multiple Unicode code points joined by zero-width joiners (U+200D). It consists of: man emoji (U+1F468), zero-width joiner, woman emoji (U+1F469), zero-width joiner, girl emoji (U+1F467), zero-width joiner, boy emoji (U+1F466). This sequence is rendered as a single family picture by emoji-aware rendering systems. To a human reader, it is one character. To a tokenizer, it is a complex byte sequence that might expand into ten, fifteen, or more tokens depending on how the tokenizer handles the constituent parts.

The semantics of emoji are also compositional in ways that tokenizers cannot represent. The flag emoji 🇺🇸 is composed of two regional indicator characters (U+1F1FA and U+1F1F8) that are paired to form the US flag. Skin tone modifiers (U+1F3FB through U+1F3FF) can be appended to many base emoji to change their displayed skin tone. Gender modifiers can be combined with profession emoji. These compositional rules are meaningful to users, but they are invisible to tokenizers that see only byte sequences. The model must learn emoji semantics entirely from exposure to examples, without any structural signal from the tokenizer.

Unicode allows the same visual text to be encoded in multiple ways through the use of combining characters. The letter "é" (e with acute accent) can be represented as a single precomposed code point (U+00E9) or as two code points: the base letter "e" (U+0065) followed by the combining acute accent character (U+0301). To any human reader, both representations display identically. But to a tokenizer, they are different character sequences.

If the tokenizer's vocabulary was built on text that used precomposed forms (as many Western European language texts do), then it will have tokens for "é" as a unit. When it encounters the decomposed form, it might not find "é" in its vocabulary and must represent the two code points separately, potentially fragmenting surrounding text as well. Two documents with identical visual content can produce different token sequences simply because they were encoded with different Unicode normalization forms.

This problem is particularly acute for diacritical marks common in French, Spanish, Portuguese, Vietnamese, and many other languages. It affects tokenization quality for these languages in ways that are hard to detect without explicit testing, because the visual output of the tokenized text looks correct even when the tokenization is inconsistent.

Unicode normalization before tokenization mitigates this problem. Applying NFKC normalization converts text to a canonical form where all characters that can be precomposed are precomposed, compatibility equivalents are unified, and the resulting encoding is consistent. Most production tokenization pipelines include this step. The HuggingFace tokenizers library applies normalization automatically for many tokenizer types. But not all pipelines apply normalization consistently, and text from diverse sources may contain mixed normalization forms that slip through.

Unicode includes many characters that are invisible to human readers: zero-width spaces (U+200B), zero-width non-joiners (U+200C), zero-width joiners (U+200D), soft hyphens (U+00AD), byte-order marks (U+FEFF), and bidirectional control characters (U+202A through U+202E and U+2066 through U+2069). These characters affect text rendering, layout, or directionality without being visible as characters in the text.

A tokenizer sees these invisible characters as real characters that affect tokenization. A zero-width space inserted between the letters of a word splits what would otherwise be a single character sequence into two separate sequences. If the tokenizer would have produced the token ["running"] for the word "running", inserting a zero-width space as "run\u200Bning" causes the tokenizer to split across the invisible character, potentially producing ["run", "ning"] or other fragments. The human-readable text still displays as "running". The tokenizer sees a different character sequence.

This invisibility has practical implications for security. An attacker can insert zero-width characters into text to manipulate tokenization without changing the text's visible appearance. If a content moderation system operates at the token level (matching against tokens representing harmful keywords), inserting invisible characters can cause the harmful keyword to split across tokens, potentially escaping detection. The human reviewer who reads the submitted text sees nothing unusual. The tokenizer sees a different sequence than the moderation system expects.

Soft hyphens (U+00AD, also called shy hyphens) are another source of invisible manipulation. They are intended to indicate optional line-break points in long words. Most rendering systems display them only when a line break occurs at that point. But a tokenizer sees them as characters and may split tokens at their position. The result, again, is a tokenization different from what the visible text would suggest.

Some tokenization failures are not about specific input types but about systematic artifacts introduced by the tokenization process itself. These artifacts arise from the interaction between training data distributions, vocabulary construction, and the specific merge rules or probability models learned during tokenizer training. They affect text across all domains and languages, not just in the special cases we have discussed so far.

Understanding these artifacts is important for building robust NLP pipelines, because they mean that small, seemingly innocuous changes to text can produce large and unexpected changes in tokenization. A system that treats text as continuous and smoothly varying is wrong: the space of tokenized representations is highly discontinuous, and nearby strings in character space can be far apart in token space.

Most subword tokenizers use whitespace as a strong signal for token boundaries. They distinguish between a token that appears at the start of a word (preceded by a space) and the same character sequence that appears mid-word. The GPT-2 tokenizer, for example, represents " the" (with leading space) as a different token than "the" (without leading space). This design reflects the fact that most English words are preceded by a space in prose, and including the leading space as part of the token helps the tokenizer identify word boundaries reliably.

But this design choice creates boundary artifacts. The same word, depending on whether it appears at the beginning of a sentence, immediately after unusual punctuation, or within a compound word, may receive different tokenizations. A word at the start of a sentence has no preceding space. A word after an opening parenthesis might have its leading space stripped during preprocessing. A word that is part of a hyphenated compound appears without a space between its components. In all these cases, the tokenizer might produce a different token for the same word root, requiring the model to learn that these different tokens carry similar meanings.

More subtly, punctuation marks and their adjacent text can create tokenization irregularities. A period followed by a capital letter (end of sentence) might be tokenized differently than a period followed by a digit (decimal number) or a period followed by a lowercase letter (abbreviation). The tokenizer applies its merge rules to the character sequence, and the result depends on what character sequences appeared frequently in training data.

Capitalization interacts with tokenization in ways that can be surprising. Many subword tokenizers are case-sensitive, meaning "bank" and "Bank" are treated as different token sequences. For BERT's cased vocabulary, both appear as separate vocabulary entries, so they tokenize identically to distinct tokens. For GPT-2, which learns its vocabulary from web text, the behavior depends on which forms were frequent enough to enter the vocabulary.

For proper nouns, this creates a persistent asymmetry. "Amazon" (the company) might receive a single token because it appears frequently in web text. "amazon" (the river, or used as an adjective) might tokenize differently if the lowercase form was less common in training data. Named entities that are rare in training data may fragment in ways that distinguish them from common words of similar spelling, potentially making it harder for the model to learn their names and properties.

The capitalization effect extends to tokenizing sentences. The first word of a sentence is capitalized but semantically identical to its lowercase form. A model that sees "The" and "the" as different tokens must learn from context that they are equivalent, which happens through training but adds noise to the learning signal. Case-insensitive tokenizers avoid this problem but lose potentially useful information about named entities and sentence boundaries that capitalization carries.

One subtle but practically important artifact is that a minor change to text can cascade into large tokenization changes. Adding one character to a word can shift the merge decisions for the entire word, potentially changing all of its tokens. This sensitivity means that tokenized text has a complex, non-local relationship to its raw string form.

Think of BPE merge rules as applied in order: the algorithm takes the pair of adjacent tokens with the highest frequency and merges them. A small change to a character sequence can cause a different pair to become the highest-frequency pair early in the merge process, which then affects all subsequent merge decisions for the surrounding sequence. The final tokenization can be completely different even though the input differs by only one character.

For robustness research, this sensitivity is a fundamental concern. Two strings that differ by a typo (a single character substitution or insertion) are semantically nearly identical to human readers. But they can tokenize completely differently, producing very different model inputs. If the model's output is sensitive to the specific token sequence it receives, a single-character typo can produce a dramatically different output. Testing models for robustness to minor perturbations should include examining how those perturbations affect tokenization.

For adversarial attacks, the same property is exploited deliberately. An attacker who understands tokenization can find character-level perturbations that produce specific desired token sequences. If the attacker wants a harmful keyword to not appear as a recognizable token, they can insert or modify characters to cause the keyword to fragment at a token boundary. The resulting text reads as the same harmful content to a human but produces a different token sequence that might escape detection.

The gap between human-readable text and tokenizer-produced token sequences opens a category of adversarial attacks that exploit this mismatch specifically. Unlike adversarial examples in vision, which require computational search for perturbations that fool classifiers, many tokenization-based attacks are simple: they rely on basic properties of Unicode encoding that are easy to apply once understood. The attacker does not need sophisticated optimization. They need knowledge of how the tokenizer behaves on specific character types.

This makes tokenization-based attacks more accessible and more pervasive than many other attack categories. A malicious user who discovers that inserting a zero-width space between the characters of a prohibited keyword causes that keyword to evade a token-based filter does not need machine learning expertise. They need to know that zero-width spaces exist and that they affect tokenization. This knowledge is increasingly widespread.

Prompt injection attacks sometimes exploit tokenization to hide control instructions from human reviewers while ensuring the model receives and executes those instructions. Consider a scenario where a content review system inspects text before it reaches a language model. If the review system operates on human-readable text and the injection is composed of characters that are invisible to humans but visible to the tokenizer, the injection might pass the human review and reach the model.

By inserting invisible Unicode characters between the characters of a harmful command, an attacker can make the command appear as innocuous text to human reviewers while ensuring the tokenizer still produces tokens that represent the command to the model. If the model has learned to follow instructions embedded in its input, and if the tokenized instruction is recognizable enough for the model to parse, the attack succeeds. The key is that the tokenizer sees the raw character sequence, including invisible characters, while the human reviewer sees only the rendered visual representation.

Conversely, attackers can use homoglyphs: characters that look visually identical to common ASCII letters but are different Unicode code points from a different block. Cyrillic а (U+0430, Cyrillic small letter a) looks indistinguishable from Latin a (U+0061) in most fonts. The string аdmin with a Cyrillic а looks exactly like admin with a Latin a. A filter that checks for the exact ASCII string admin will not match the Cyrillic version. If the model's behavior is triggered by a token that happens to be shared (because BPE sees the byte sequence and the byte values are the same), or if the model has learned to treat visually similar tokens as equivalent, the homoglyph might still achieve the attacker's goal while evading ASCII-based filters.

Some jailbreak attempts exploit the fact that a harmful keyword split across token boundaries might not be recognized by a classifier trained on token sequences. If a safety classifier was trained to detect the token "bomb" but the tokenizer in a particular context splits the word as ["bo", "mb"] or ["b", "omb"], the classifier may not activate. The harmful intent is present in the text, and a human reader would recognize it, but the token sequence that the classifier sees does not match the pattern it was trained to detect.

This attack is more reliable than it might seem, because tokenization is context-sensitive. The same word can tokenize differently depending on what precedes it, what case it appears in, and whether it appears with or without a preceding space. An attacker who probes a system can discover which tokenization context causes a keyword to split favorably, then craft inputs that reliably produce that context.

More sophisticated jailbreak techniques encode instructions in transformed forms that the model can decode but that automated systems may not recognize. Reversing text character by character, encoding with simple substitution ciphers, using Pig Latin, spacing out letters, or representing text in a programming language that evaluates to the harmful string are all techniques that have been demonstrated in practice. The effectiveness of these approaches depends on whether the underlying model has learned to process such transformations (which capable models often have, through training on diverse text) and on whether the safety systems can recognize the encoding.

The key insight is that safety systems that operate at the token level are inherently limited by the tokenizer's behavior. If the attacker can manipulate which tokens are produced from a given piece of text, they can potentially manipulate whether safety classifiers activate. This motivates safety approaches that operate at multiple levels: token-level classification, character-level normalization, and semantic understanding of meaning regardless of surface form.

Defending against adversarial tokenization requires understanding the full attack surface and applying mitigations at multiple points in the pipeline. No single defense is sufficient. The most robust approach layers multiple strategies.

Unicode normalization is the foundational defense. Applying NFKC normalization before tokenization collapses homoglyph variants (Cyrillic and Latin characters that appear identical are both mapped to their canonical forms), resolves combining character alternatives, and eliminates compatibility ambiguities. This substantially reduces the surface area for homoglyph attacks and encoding-based confusion, though it does not eliminate all possibilities because not all homoglyphs are Unicode equivalents.

Invisible character stripping removes zero-width spaces, zero-width non-joiners, soft hyphens, byte-order marks, and bidirectional control characters that can be inserted to manipulate tokenization without affecting visual appearance. This is a straightforward preprocessing step that eliminates a whole class of injection attacks. The characters being stripped serve no legitimate purpose in most NLP applications, so their removal does not damage the content.

Input validation can reject text containing characters outside the expected range for the application's use case. A customer service chatbot that expects English text can reject inputs containing Cyrillic or Arabic characters (unless the application explicitly supports those languages). A code completion tool can validate that inputs contain only characters that appear in source code. These constraints reduce the attack surface by preventing the input of exotic Unicode constructions that only make sense in an adversarial context.

Ensemble detection uses multiple tokenization schemes in parallel. If a safety classifier is trained on one tokenizer's output and an attacker exploits that tokenizer's behavior to produce a non-triggering token sequence, a second classifier trained on a different tokenizer's output might still detect the harmful content. Combining the outputs of classifiers with different tokenization schemes makes it substantially harder for an attacker to find a single input that simultaneously evades all of them, because the input would need to exploit different vulnerabilities in multiple tokenizers at once.

Evaluating tokenization quality is not as simple as measuring vocabulary size or compression ratio. Good tokenization is domain-specific: the right tokenizer for English prose is not the right tokenizer for Python code or Arabic medical records. Several quantitative metrics help assess tokenization quality for specific applications, and understanding these metrics helps you make informed choices when selecting or evaluating tokenizers.

The key insight about tokenization evaluation is that the metrics that matter depend on what you plan to do with the tokenized text. A tokenizer that achieves excellent fertility for English might be terrible for Finnish. A tokenizer that handles natural language well might be poor for source code. Measuring the right things for your specific application is essential for making good tokenizer choices.

Fertility, measured as tokens per word (or tokens per character for scripts without spaces), is the most direct and interpretable measure of tokenization efficiency. It tells you how much of your context window a given amount of text will consume. Ideal fertility for a well-handled language is between 1.0 and 1.5 tokens per word: each word gets roughly one token on average, with some words receiving two tokens for their morphological components.

Fertility degrades gracefully with linguistic distance from the training corpus. Languages closely related to English, sharing vocabulary, morphology, and scripts, achieve fertility of 1.5 to 2.0. More distant languages may reach 4.0 to 8.0. These numbers matter practically because they directly translate to effective context window sizes. An Arabic user who needs fertility of 6.0 to encode Arabic text effectively has a context window six times smaller in semantic terms than an English user with fertility 1.0. For a fixed inference budget, this means the Arabic user can ask shorter questions, receive shorter answers, and benefit from less contextual grounding.

Fertility should be measured separately for each language and domain of interest, using representative text samples. A tokenizer might have excellent fertility for formal literary English (1.1) and poor fertility for Twitter English with emoji and informal spellings (1.8), making it suboptimal for social media applications. Measuring on the right domain is necessary for making accurate predictions about real-world performance.

A tokenizer that splits words at morphologically meaningful boundaries produces representations that are easier for models to learn from. Morphological alignment measures how closely the tokenizer's splits correspond to linguistically meaningful morpheme boundaries.

Consider the word "running". A tokenizer might split it as ["run", "##ning"] (WordPiece style, where ##ning signals a suffix), or as ["runn", "##ing"], or as a single token ["running"] if the word appears frequently enough. The split ["run", "##ning"] is morphologically aligned: run is the stem and ##ning is a suffix. The model can potentially learn the relationship between run, running, runner, and runs more easily because the shared stem run appears consistently across their tokenizations. The split ["runn", "##ing"] has no morphological meaning, because runn is not a morpheme in English.

Morphological alignment can be measured by comparing tokenizer splits against gold-standard morphological analyses from linguistic resources such as MorphyNet or the Universal Dependencies treebanks. High overlap between tokenizer splits and morpheme boundaries correlates with better downstream performance on tasks that require understanding morphological relationships: part-of-speech tagging, lemmatization, morphological analysis, and semantic similarity across inflected forms.

For morphologically complex languages like Finnish, Turkish, or Hungarian, where a single word root can appear with dozens of different suffixes, morphological alignment becomes especially important. A tokenizer that correctly identifies the shared roots across inflected forms allows the model to learn generalizable morphological patterns. A tokenizer that fragments words at arbitrary boundaries must learn each surface form independently, requiring more parameters and more training examples.

Ideally, tokens should correspond to units of meaning. A token that always appears as part of longer meaningful units (like ##tion, a common English noun suffix) has low standalone semantic coherence, which is appropriate: it is a morphological piece that gains meaning only in combination. But a token that appears in wildly different semantic contexts might be grouping semantically unrelated character sequences together, which makes it harder for the model to learn a consistent representation.

Measuring semantic coherence quantitatively requires evaluating the consistency of each token's contextual usage. Tokens that appear in similar contexts across many documents should have higher semantic coherence than tokens that appear in very different contexts. This can be measured using distributional statistics: the entropy of a token's contextual distribution (high entropy means the token appears in diverse contexts, suggesting low coherence) or the mutual information between a token and its neighboring tokens (higher mutual information suggests more consistent contextual usage).

Semantic coherence analysis can identify tokens that are functioning as compression artifacts rather than meaningful units, informing decisions about vocabulary size and tokenizer design. Vocabularies with many low-coherence tokens are candidates for reduction: replacing those tokens with their constituent byte-level representations might improve model learning without reducing compression efficiency significantly.

The most direct and rigorous measure of tokenization quality is downstream task performance. Two tokenizers can be compared by training the same model architecture with each tokenizer on identical data and measuring performance on a held-out test set of task-relevant examples. Better tokenization for a given task and language will produce higher task performance, all else being equal.

This approach is expensive: it requires training full models rather than just evaluating the tokenizers themselves. But it is definitive in a way that proxy metrics are not. Fertility, morphological alignment, and semantic coherence are all approximations of the true goal, which is to produce a representation that allows the model to learn and perform well. Downstream task evaluation measures the true goal directly.

When researchers evaluate tokenizers for specific domains like biomedical text, legal documents, or source code, they typically use downstream task performance on domain-specific benchmarks to select the best tokenizer. The findings often confirm what fertility and morphological alignment predict, but sometimes reveal counterintuitive results: a tokenizer with slightly higher fertility might outperform one with lower fertility because its splits better preserve semantic units for the specific task.

Let's work through these tokenization challenges with concrete examples using the HuggingFace tokenizers library. We'll examine how GPT-2's BPE tokenizer handles numbers, code, multilingual text, and emoji, observing each failure mode directly.

Let's see how numbers of different sizes and formats tokenize. We expect to observe single tokens for small, frequent numbers and fragmentation for larger, rarer ones.

The expression 9999 + 1 = 10000 reveals the carry propagation problem. The model must process 9999 as either a single token or multiple fragments, then produce 10000 as its output. Because 10000 has a different tokenization than 9999 + 1, the model cannot implement a simple character-level carry algorithm. It must have learned the relationship between the input and output token sequences from examples.

Let's examine how Python code fragments when processed by a tokenizer trained on English prose:

calculate_moving_average gets split at non-semantic boundaries. The tokenizer has no knowledge that shows mark word boundaries in Python identifiers. The camelCase identifier getUserPreferences also fragments at points that do not align with the conceptual components get, User, and Preferences. Each fragment must be processed as a separate token by the model. This consumes context budget and potentially obscuring the relationship between semantically related identifiers.

Let's measure fertility across different languages using semantically equivalent content. We use "The quick brown fox jumps over the lazy dog" translated into multiple languages, which gives us comparable semantic content for the comparison.

The fertility disparity is stark. English achieves near-ideal fertility because the tokenizer was trained on English text and has efficient vocabulary coverage for English words. Non-Latin languages require far more tokens to represent equivalent content. This means non-English users effectively have a smaller context window when using English-trained tokenizers. For a model with a 4,096-token budget, the practical difference between English and Arabic fertility translates to a real difference in how much information each user can include in a prompt.

The emoji examples reveal how dramatically token counts can vary. The simple snake emoji 🐍 requires multiple byte-level tokens because it is a non-ASCII code point encoded as four bytes in UTF-8. The family emoji 👨‍👩‍👧‍👦, which looks like a single character in any emoji-aware display, requires far more tokens because it is a sequence of multiple code points joined by zero-width joiners. From the tokenizer's perspective, this "single character" is a complex byte sequence that may expand into ten or more tokens. Math symbols and mixed-script text also reveal the byte-level fallback mechanism that handles characters without dedicated vocabulary entries.

Let's demonstrate how invisible Unicode characters and homoglyphs can produce different tokenizations from visually identical text:

The homoglyph example demonstrates that visually identical strings can produce different token sequences. The zero-width space example shows how a completely invisible character can split what would otherwise be a single token into two. These differences are exploitable: a filter that operates on token sequences can be evaded by inserting the right invisible character to cause the target word to split at a boundary the filter does not check.

The bar chart reveals a clear pattern: numbers that appeared frequently in the training corpus receive single tokens. Larger or less common numbers fragment into multiple tokens, disrupting the positional structure needed for arithmetic. Notice that the fragmentation is not monotone: 999 might receive more tokens than 1000 if 1000 appeared more frequently in training data as a round number. This frequency-based inconsistency is a fundamental property of the BPE algorithm when applied to numeric text.

Working with tokenizers for robustness and quality involves these important considerations:

• add_prefix_space: Many tokenizers treat the first token of a sequence differently (without a leading space). Setting add_prefix_space=True ensures consistent tokenization regardless of position in a longer text.
• truncation and max_length: Controls how long inputs are handled. High-fertility languages will hit length limits faster than English for the same semantic content.
• padding: Required for batch processing but consumes token budget. The padding length is determined by the longest sequence in the batch, which for high-fertility languages will be longer.
• Normalization settings: In SentencePiece and HuggingFace tokenizers, normalization rules (NFKC, NFC) affect how Unicode is handled before tokenization. Enabling NFKC normalization is the most important defense against Unicode-based attacks.

Tokenization challenges have practical consequences. They shape which problems language models can solve, which users they serve well, how robust they are to adversarial manipulation, and ultimately which communities benefit from the technology and which are left behind. Understanding these limitations is essential for responsible deployment of language models.

The number tokenization problem has direct practical consequences for any application involving numeric reasoning. Financial analysis that asks a model to reason about prices, returns, or statistical quantities requires the model to handle fragmented numeric representations. Scientific computation where precise numeric values matter becomes unreliable when those values tokenize inconsistently. Date arithmetic, where the model must reason about days, months, and years, requires understanding numeric relationships that tokenization obscures. Calendar and time-zone reasoning, increasingly important in applications that interact with users globally, requires correct interpretation of fragmented numeric sequences.

Models that receive fragmented numeric representations must compensate through parameter count and training data. They need more capacity to represent implicit numeric relationships, and they need more training examples of each numeric operation to learn those relationships from indirect evidence. This is one reason why purpose-built tools like Python code interpreters, calculators, and SQL query engines often outperform raw language model inference on multi-step calculations. The tool eliminates the tokenization problem by operating on numbers directly. For new applications involving numeric reasoning, building tool use into the architecture from the start is often more reliable than hoping the model's implicit numeric representations are sufficient.

For multilingual applications, the fertility disparity is a form of inequality embedded in infrastructure decisions. The gap between English fertility and non-English fertility is not a law of nature. It is a consequence of training tokenizers predominantly on English text. Dedicated multilingual tokenizers trained with balanced language corpora, or byte-level tokenizers that treat all scripts equitably at the byte level, substantially reduce but rarely eliminate the disparity entirely. The models that have made the most progress on multilingual equity are those that explicitly included balanced multilingual data at the tokenizer training stage, not just at the language model training stage.

Code tokenization quality has become a practical bottleneck as language models are increasingly deployed for software engineering tasks. The fragmentation of long identifiers and the handling of whitespace-sensitive languages like Python, YAML, and Markdown affect how well models can learn the structure of programs. Purpose-trained code models like CodeLlama and StarCoder use tokenizers with richer code vocabularies, reducing identifier fragmentation and improving the model's ability to learn programming patterns. For teams building coding assistants, the choice of tokenizer has measurable impact on benchmark performance and user satisfaction, not just on token efficiency.

Adversarial tokenization represents an active and evolving research frontier. As language models are deployed in security-sensitive contexts (content moderation, customer service, legal advice, medical triage), adversarial inputs that exploit tokenizer behavior become a real attack surface with real consequences. Defenses that normalize input at the Unicode level, validate character ranges, and test models against known tokenization-based evasion strategies are becoming part of responsible deployment practice. This is not optional for high-stakes applications. The attack techniques are well documented and actively used. Deploying a language model in a content-moderation capacity without Unicode normalization and invisible character stripping is leaving a known vulnerability open.

Perhaps the most important insight this chapter offers is that tokenization is not a solved problem, and it is not a neutral technical choice. It is a design decision with significant downstream consequences that are not evenly distributed across languages, domains, and use cases. The decision to train a tokenizer primarily on English text, which was made for practical reasons of data availability, encodes a particular set of tradeoffs: excellent English performance, poor non-English performance, and a complex interaction with numeric and code content. Understanding these tradeoffs clearly is the first step toward making better decisions about which tokenizer to use, how to preprocess input, and how to evaluate models for fairness across different user populations.

This chapter examined the places where subword tokenization breaks down and the mechanisms underlying each failure:

• Number tokenization fragments numeric values based on corpus frequency rather than numeric structure, harming arithmetic reasoning. The fundamental problem is that statistical co-occurrence cannot capture the positional value system that gives numbers meaning. Digit-level tokenization, where each digit receives its own token, is a principled solution at the cost of higher token counts. For applications requiring precise arithmetic, external computation tools are often more reliable than implicit numeric reasoning.

• Code tokenization splits identifiers at non-semantic boundaries, handles whitespace-sensitive indentation based on natural language patterns, and produces redundant token sequences for common code structures. Code-specific tokenizers with richer vocabularies improve this substantially. The gap in context window efficiency between general-purpose and code-specialized tokenizers is measurable and directly affects performance on longer programs.

• Multilingual fertility disparity gives non-English languages higher token counts per unit of semantic content, reducing effective context window size and increasing computational cost. This disparity is a form of structural inequality in how different language communities experience language models. Balanced multilingual training corpora, dedicated multilingual tokenizers, and byte-level approaches partially address this but rarely eliminate it entirely.

• Emoji and Unicode edge cases arise from the complexity of Unicode encoding, combining characters, zero-width joiners, and the diversity of scripts. A single emoji that looks like one character can expand into dozens of tokens. Normalization before tokenization mitigates the combining-character ambiguity but cannot eliminate all sources of complexity in modern text.

• Tokenization artifacts including boundary sensitivity, capitalization effects, and cascading changes from single-character perturbations affect model robustness to minor input variations. These artifacts mean that the space of tokenized representations is highly discontinuous: nearby strings in character space can be far apart in token space.

• Adversarial tokenization exploits the gap between human-readable text and token sequences through homoglyphs, invisible characters, and encoding tricks. These attacks are practical and actively used. Defense requires multiple layers: Unicode normalization, invisible character stripping, input validation, and ensemble classifiers.

• Measuring tokenization quality requires domain-specific evaluation of fertility, morphological alignment, semantic coherence, and downstream task performance. Vocabulary size and compression ratio alone do not capture quality for specific use cases.

These challenges motivate ongoing research into better tokenization schemes: byte-level approaches that treat all scripts equitably, character-level models that sidestep the statistical subword framework entirely, and learned tokenizers that optimize for specific downstream tasks rather than raw compression. In the next part of the book, we move from tokenization to the neural architectures that consume these token sequences, building on the understanding of tokenization's strengths and limitations to interpret how those architectures work and why they perform as they do.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about tokenization challenges.