Tokenization Challenges › Summary

This chapter examined the places where subword tokenization breaks down and the mechanisms underlying each failure:

• Number tokenization fragments numeric values based on corpus frequency rather than numeric structure, harming arithmetic reasoning. The fundamental problem is that statistical co-occurrence cannot capture the positional value system that gives numbers meaning. Digit-level tokenization, where each digit receives its own token, is a principled solution at the cost of higher token counts. For applications requiring precise arithmetic, external computation tools are often more reliable than implicit numeric reasoning.

• Code tokenization splits identifiers at non-semantic boundaries, handles whitespace-sensitive indentation based on natural language patterns, and produces redundant token sequences for common code structures. Code-specific tokenizers with richer vocabularies improve this substantially. The gap in context window efficiency between general-purpose and code-specialized tokenizers is measurable and directly affects performance on longer programs.

• Multilingual fertility disparity gives non-English languages higher token counts per unit of semantic content, reducing effective context window size and increasing computational cost. This disparity is a form of structural inequality in how different language communities experience language models. Balanced multilingual training corpora, dedicated multilingual tokenizers, and byte-level approaches partially address this but rarely eliminate it entirely.

• Emoji and Unicode edge cases arise from the complexity of Unicode encoding, combining characters, zero-width joiners, and the diversity of scripts. A single emoji that looks like one character can expand into dozens of tokens. Normalization before tokenization mitigates the combining-character ambiguity but cannot eliminate all sources of complexity in modern text.

• Tokenization artifacts including boundary sensitivity, capitalization effects, and cascading changes from single-character perturbations affect model robustness to minor input variations. These artifacts mean that the space of tokenized representations is highly discontinuous: nearby strings in character space can be far apart in token space.

• Adversarial tokenization exploits the gap between human-readable text and token sequences through homoglyphs, invisible characters, and encoding tricks. These attacks are practical and actively used. Defense requires multiple layers: Unicode normalization, invisible character stripping, input validation, and ensemble classifiers.

• Measuring tokenization quality requires domain-specific evaluation of fertility, morphological alignment, semantic coherence, and downstream task performance. Vocabulary size and compression ratio alone do not capture quality for specific use cases.

These challenges motivate ongoing research into better tokenization schemes: byte-level approaches that treat all scripts equitably, character-level models that sidestep the statistical subword framework entirely, and learned tokenizers that optimize for specific downstream tasks rather than raw compression. In the next part of the book, we move from tokenization to the neural architectures that consume these token sequences, building on the understanding of tokenization's strengths and limitations to interpret how those architectures work and why they perform as they do.