Word Tokenization › Language-Specific Challenges

English, with its space-separated words, is relatively easy to tokenize. Other languages present unique challenges that reveal how much the whitespace convention assumes. When we step outside of languages that use spaces, we realize that tokenization is not a universal procedure but a language-specific task that must be designed with the particular language's writing system and morphology in mind.

The diversity of human writing systems is remarkable. Some scripts separate words with spaces (Latin, Cyrillic, Arabic). Others run characters together without any delimiters (Chinese, Japanese for many purposes). Some languages build words by concatenating morphemes into long compound words (German, Finnish, Turkish). Some have extremely complex morphological systems where a single word can encode what English expresses in a whole sentence. There is no single tokenization algorithm that handles all of these correctly.

Chinese text contains no spaces between words. Characters flow continuously, and determining word boundaries requires linguistic knowledge. This is not a defect of the writing system; Chinese characters are ideographic and have been written without spaces for thousands of years. The concept of a "word" in Chinese is itself somewhat different from the English concept.

Consider the string "我喜欢学习自然语言处理" (I like studying natural language processing). There are eleven characters, but how many words? The answer depends on how you define "word". If you count morphemes, you get more. If you count content units, you might count differently. The standard NLP answer is to try to segment the string into units that correspond to dictionary entries, which requires a combination of dictionary lookup, statistical modeling, and contextual reasoning.

Character tokenization produces 11 tokens, one per character. Word tokenization produces fewer, more meaningful units. The word tokenizer identifies "自然语言处理" (natural language processing) as a compound term and keeps it together, while character tokenization would split it into six individual characters. This matters: "自然" alone means "nature" or "natural", but the full six-character string has a technical meaning specific to the field of NLP.

The impact of tokenization granularity varies dramatically across languages:

Languages without explicit word boundaries (Chinese, Japanese) show smaller differences between character and word tokenization because word boundaries must be inferred. English shows the largest ratio because whitespace already provides clear word boundaries, making character tokenization produce many more tokens than word tokenization.

Japanese uses three writing systems in combination: hiragana (a syllabic alphabet for native Japanese words and grammatical elements), katakana (a syllabic alphabet for foreign words and emphasis), and kanji (Chinese-derived ideographic characters). Words can be written in any combination of these scripts, and spaces are rarely used. This creates a tokenization challenge that is more complex than Chinese because the tokenizer must handle missing spaces and script changes from character to character.

The mixing of scripts provides useful cues. Foreign words written in katakana form natural units. Sequences of kanji interrupted by hiragana often indicate word boundaries. Latin letters embedded in Japanese text (like "NLP" or "Python") form their own units. A good Japanese tokenizer exploits these cues while also relying on a morphological dictionary and statistical models for the many cases where script alone is ambiguous.

Japanese tokenization must handle the mixing of scripts. "Python" and "NLP" are written in Latin characters, while the rest uses Japanese scripts. A good tokenizer recognizes these boundaries and segments appropriately. Notice that even at the "word" level, the tokenization is quite fine-grained: grammatical particles like "は" (topic marker) and "を" (object marker) become separate tokens, because in Japanese, these particles are separate morphemes, not inflectional endings attached to the preceding word.

German creates long compound words by concatenating shorter words without spaces. "Donaudampfschifffahrtsgesellschaftskapitän" (Danube steamship company captain) is a single orthographic word consisting of four meaningful components. This is not an edge case or a quirk: German compounds are extremely common and productive. New compounds are formed freely by combining existing words.

For most NLP tasks, you have a choice: treat the compound as a single token, or decompose it into its parts. The right choice depends on the task. For information retrieval, treating "Krankenhaus" (hospital) as a single token means a query for "Kranken" (sick) won't find documents about hospitals. Decomposing it allows the match, at the cost of potentially matching unrelated words that happen to contain "Kranken" as a component.

For some applications, you might want to decompose compounds into their constituent parts. This requires morphological analysis beyond simple tokenization. Modern NLP approaches often sidestep this problem by using subword tokenization methods like BPE that learn to split words at frequently occurring boundaries, which often coincide with morpheme boundaries in languages like German.

Arabic presents multiple challenges: right-to-left script, complex morphology with prefixes and suffixes attached to words as clitics, and optional vowel diacritics. A single Arabic word can pack what English might express as a full sentence. The word "وسيكتبونها" means "and they will write it", combining a conjunction ("و", and), a future marker ("س"), the verb root (write), a plural subject marker, and a feminine object pronoun, all without spaces.

This morphological richness means that whitespace tokenization, while technically possible for Arabic, produces tokens that are linguistically very coarse. The tokens you get are morphologically complex forms that include clitics, and most NLP tasks benefit from separating those clitics from the stem they attach to.