Tokenization Challenges › Tokenization Artifacts

Some tokenization failures are not about specific input types but about systematic artifacts introduced by the tokenization process itself. These artifacts arise from the interaction between training data distributions, vocabulary construction, and the specific merge rules or probability models learned during tokenizer training. They affect text across all domains and languages, not just in the special cases we have discussed so far.

Understanding these artifacts is important for building robust NLP pipelines, because they mean that small, seemingly innocuous changes to text can produce large and unexpected changes in tokenization. A system that treats text as continuous and smoothly varying is wrong: the space of tokenized representations is highly discontinuous, and nearby strings in character space can be far apart in token space.

Most subword tokenizers use whitespace as a strong signal for token boundaries. They distinguish between a token that appears at the start of a word (preceded by a space) and the same character sequence that appears mid-word. The GPT-2 tokenizer, for example, represents " the" (with leading space) as a different token than "the" (without leading space). This design reflects the fact that most English words are preceded by a space in prose, and including the leading space as part of the token helps the tokenizer identify word boundaries reliably.

But this design choice creates boundary artifacts. The same word, depending on whether it appears at the beginning of a sentence, immediately after unusual punctuation, or within a compound word, may receive different tokenizations. A word at the start of a sentence has no preceding space. A word after an opening parenthesis might have its leading space stripped during preprocessing. A word that is part of a hyphenated compound appears without a space between its components. In all these cases, the tokenizer might produce a different token for the same word root, requiring the model to learn that these different tokens carry similar meanings.

More subtly, punctuation marks and their adjacent text can create tokenization irregularities. A period followed by a capital letter (end of sentence) might be tokenized differently than a period followed by a digit (decimal number) or a period followed by a lowercase letter (abbreviation). The tokenizer applies its merge rules to the character sequence, and the result depends on what character sequences appeared frequently in training data.

Capitalization interacts with tokenization in ways that can be surprising. Many subword tokenizers are case-sensitive, meaning "bank" and "Bank" are treated as different token sequences. For BERT's cased vocabulary, both appear as separate vocabulary entries, so they tokenize identically to distinct tokens. For GPT-2, which learns its vocabulary from web text, the behavior depends on which forms were frequent enough to enter the vocabulary.

For proper nouns, this creates a persistent asymmetry. "Amazon" (the company) might receive a single token because it appears frequently in web text. "amazon" (the river, or used as an adjective) might tokenize differently if the lowercase form was less common in training data. Named entities that are rare in training data may fragment in ways that distinguish them from common words of similar spelling, potentially making it harder for the model to learn their names and properties.

The capitalization effect extends to tokenizing sentences. The first word of a sentence is capitalized but semantically identical to its lowercase form. A model that sees "The" and "the" as different tokens must learn from context that they are equivalent, which happens through training but adds noise to the learning signal. Case-insensitive tokenizers avoid this problem but lose potentially useful information about named entities and sentence boundaries that capitalization carries.

One subtle but practically important artifact is that a minor change to text can cascade into large tokenization changes. Adding one character to a word can shift the merge decisions for the entire word, potentially changing all of its tokens. This sensitivity means that tokenized text has a complex, non-local relationship to its raw string form.

Think of BPE merge rules as applied in order: the algorithm takes the pair of adjacent tokens with the highest frequency and merges them. A small change to a character sequence can cause a different pair to become the highest-frequency pair early in the merge process, which then affects all subsequent merge decisions for the surrounding sequence. The final tokenization can be completely different even though the input differs by only one character.

For robustness research, this sensitivity is a fundamental concern. Two strings that differ by a typo (a single character substitution or insertion) are semantically nearly identical to human readers. But they can tokenize completely differently, producing very different model inputs. If the model's output is sensitive to the specific token sequence it receives, a single-character typo can produce a dramatically different output. Testing models for robustness to minor perturbations should include examining how those perturbations affect tokenization.

For adversarial attacks, the same property is exploited deliberately. An attacker who understands tokenization can find character-level perturbations that produce specific desired token sequences. If the attacker wants a harmful keyword to not appear as a recognizable token, they can insert or modify characters to cause the keyword to fragment at a token boundary. The resulting text reads as the same harmful content to a human but produces a different token sequence that might escape detection.