Word Tokenization › Limitations and Challenges

Word tokenization, despite decades of research, remains imperfect. Understanding its limitations informs practical choices about configuration and alternatives.

No universal definition of "word" is perhaps the most fundamental limitation. Linguistic theories disagree on what constitutes a word. In morphological theory, "running" is a single word with the base "run" and the suffix "-ing". In some traditions, each morpheme is a separate word. In others, phonological words (defined by stress patterns) differ from orthographic words (defined by spaces). Tokenizers make practical choices that may not align with any particular theory, and those choices can interact unexpectedly with downstream tasks.

Language diversity means that rules that work for English fail for Chinese, Arabic, or German. This is not a minor practical inconvenience; it is a fundamental limitation of rule-based approaches. Each language family requires specialized approaches, and building high-quality tokenizers for low-resource languages (where annotated data is scarce) is an active research challenge. The Universal Dependencies project has made progress on cross-lingual tokenization standards, but significant gaps remain.

Domain variation means that medical text, legal documents, social media, and code each have unique tokenization challenges. A tokenizer trained on news articles may struggle with tweets (which contain hashtags, @mentions, and unconventional abbreviations). A tokenizer designed for prose may mishandle mathematical expressions. A tokenizer for English may work poorly on code-switched text where speakers mix multiple languages in a single utterance.

Evolving language means that new words and abbreviations emerge while conventions change. "COVID-19", "blockchain", and emoji didn't exist decades ago. Social media has accelerated the pace of linguistic change. A tokenizer that covered common 2010 text may be incomplete for 2024 text. This is an ongoing maintenance problem, not a problem that can be solved once and forgotten.

Irreversibility is a less-discussed but important limitation. Once you tokenize text, it can be difficult or impossible to reconstruct the original exactly. If you lower-case tokens, normalize whitespace, or split contractions, you lose information. For most NLP tasks this is fine, but for tasks like machine translation or text generation, where the output needs to be valid text, this irreversibility can cause problems.