Tokenization Challenges › Multilingual Text and the Token Budget Problem

When a tokenizer is trained primarily on English text, non-English languages receive systematically worse tokenization. This causes concrete and measurable downstream effects on model performance, computational cost, and the practical usability of language models for the billions of people who do not primarily use English.

The fundamental issue is that a tokenizer's vocabulary is a finite resource. Thirty thousand or fifty thousand entries can only cover so much linguistic territory. A tokenizer trained on English text fills its vocabulary with English-relevant subwords: common English word fragments, frequent English words, English morphological patterns. When this tokenizer encounters Finnish, Arabic, Chinese, or any other language, it cannot cover those languages' patterns efficiently. It falls back to shorter, more frequent subword fragments or to individual characters, producing many more tokens to represent the same semantic content.

Think of the vocabulary as a set of compression shortcuts. If you have a shortcut for every common English word and word fragment, you can compress English text efficiently. But if you apply that same shortcut table to French text, many French words will not match any shortcut, and you will need to decompose them into smaller pieces. The same information is present, but it takes more space to encode. In the tokenizer context, more space means more tokens, which means more context window usage and more computation.

The concept of fertility measures how many tokens a given amount of text requires. Formally, the fertility of a tokenizer on a corpus is the average number of tokens produced per word (or per character for scripts without word boundaries). A tokenizer with low fertility for a language is efficient for that language: it can pack more semantic content into a fixed number of tokens. A tokenizer with high fertility for a language is inefficient: it uses many tokens to represent the same content, leaving less room in the context window for other information.

English-trained tokenizers typically achieve fertility close to 1.0 for English: roughly one token per word. For morphologically simple language like English, where words rarely have many different endings, this is achievable because common words fit in single tokens. For morphologically rich languages, where the same root can appear with dozens of different suffixes, fertility rises because the tokenizer cannot cover all the surface forms efficiently.

Consider a simple sentence in several languages. The English version tokenizes into approximately 8 tokens. The Finnish version of the same sentence might require 15 tokens because Finnish has complex morphological suffixes that are rare in an English-trained tokenizer's vocabulary. The Arabic version might require 20 tokens because Arabic uses prefixes and suffixes heavily and its script has no equivalents in English vocabulary entries. The Thai version might require even more tokens because Thai has no word boundaries, forcing the tokenizer to process the text character by character or in very short sequences.

The practical consequence of this disparity is stark. If your model has a context window of 4,096 tokens, an English speaker can fit roughly 3,000 English words into a prompt. An Arabic speaker attempting to communicate the same information in Arabic can fit perhaps 1,500 Arabic words, a fifty percent reduction in effective context. They are paying a computational and informational tax for communicating in their native language rather than English. This is not a design goal. It is an unintended consequence of training tokenizers on imbalanced corpora.

This disparity compounds in three ways:

• Shorter effective context: non-English users cannot fit as much content into a fixed context window, which means less relevant context for each query and reduced performance on tasks that benefit from long context.
• Higher computational cost: more tokens means more attention computation. At inference time, processing an Arabic sentence costs more than processing its English translation of equivalent semantic content, making the model more expensive to operate for non-English users.
• Less training signal: the same training corpus that produces a high-English vocabulary also tends to contain more English text, so non-English languages receive less training data in addition to less efficient tokenization, creating a compounding disadvantage.

Different writing systems pose categorically different tokenization challenges that go beyond simple vocabulary coverage. Latin-script languages share a 26-letter alphabet, so tokenization patterns transfer across them to some degree: subwords that appear in English also appear in French, German, and Spanish, giving those languages better coverage than completely different scripts.

Chinese presents a specific challenge because Chinese writing does not use spaces to mark word boundaries. Chinese words can be one, two, or three characters, and the same sequence of characters can be segmented as words in multiple ways depending on context. Most tokenizers handle Chinese by treating each character as a potential token unit. This produces a workable tokenization, but one that does not respect word-level semantics. Two-character compounds that function as single words in Chinese get split into individual character tokens, potentially obscuring the compound meaning.

Japanese adds another layer of complexity by mixing four different scripts in a single sentence: hiragana (phonetic syllabary for grammatical elements), katakana (phonetic syllabary for foreign words and emphasis), kanji (ideographic characters borrowed from Chinese), and often Latin characters for technical terms or brand names. A single Japanese sentence might contain all four scripts. A tokenizer must handle each script transition gracefully, correctly applying the appropriate tokenization rules for each script. Standard English-trained tokenizers typically handle Latin characters in Japanese text as if they were English words, handle kanji with byte-level fallbacks, and handle hiragana and katakana partially or inconsistently.

Arabic adds right-to-left directionality, complex cursive shaping where the appearance of a letter depends on its position in a word, and an affixation system where prefixes and suffixes fuse morphologically with roots. These properties mean that the surface forms of Arabic words are highly diverse, requiring either a very large Arabic-specific vocabulary or aggressive fragmentation. Byte-level tokenizers that process Arabic character by character or byte by byte produce tokenizations that are functional but highly inefficient.

Byte-level tokenizers offer one universal solution: they represent every possible byte as a token. This ensures that any text in any script can be tokenized regardless of its Unicode representation. But byte-level representations of non-Latin scripts are severely inefficient. A single Chinese character encoded in UTF-8 requires three bytes, so a ten-character Chinese sentence becomes thirty byte tokens. Compared to a native Chinese tokenizer that gives each character its own token, byte-level tokenization triples the context window cost. The universality of byte-level tokenization is real, but the efficiency cost for non-Latin scripts is large.

Real multilingual text frequently mixes languages within a single document or even within a single sentence. This phenomenon, known as code-switching in linguistics, is pervasive in online communication. A Spanish-English bilingual speaker might write "I was like, no puede ser, you know?" inserting a Spanish phrase into an otherwise English sentence. A technical forum in German might include English programming terminology throughout a German discussion. Social media posts in Korean might freely mix Korean and English words.

Subword tokenizers trained on multilingual corpora can handle code-switching by having vocabulary entries for both languages. When the tokenizer encounters a Spanish phrase in an English sentence, it applies Spanish-appropriate tokenization to those words and English-appropriate tokenization to the English words. This works reasonably well for neighboring languages with overlapping scripts and borrowings, less well for languages with completely different scripts where the boundary between scripts must be handled as a context switch.

The deeper challenge of code-switching is semantic. When a model encounters a Spanish phrase embedded in English text, it must understand that the Spanish phrase is semantically equivalent to the English word or phrase that would appear in its place. The tokenizer provides no signal about this semantic equivalence. The model must infer from context that "no puede ser" is functioning as an expression of disbelief, equivalent to English expressions like "no way" or "that can't be." Multilingual models trained on large amounts of code-switched text learn these patterns, but the learning is indirect and requires much more training data than a model that had explicit cross-lingual alignment signals would need.

Multilingual models like mBERT and XLM-R address the code-switching challenge by training on large multilingual corpora and learning shared representations for concepts across languages. Because they see translations and multilingual documents, they learn that "run" and "courir" and "laufen" all activate similar representations. But this learning is primarily through shared contextual patterns, not through the tokenizer's structure. The tokenizer itself does not provide cross-lingual signals. The fundamental fertility disparity between English and morphologically complex languages remains even in the most capable multilingual models.