SentencePiece › Why Pretokenization Is a Problem

Before you can understand SentencePiece's contribution, you need to understand the problem it solves. Pretokenization has been a standard step in NLP pipelines for decades, and its limitations are subtle enough that they often go unnoticed until you try to build something truly multilingual.

Traditional NLP pipelines use a process called pretokenization before applying any subword algorithm. Pretokenization is the step that splits raw text into a sequence of "words" using language-specific rules. For English, you might split on whitespace and punctuation. For German, you need to handle long compound words differently. For Chinese and Japanese, there are no spaces at all, so you need a separate word segmentation model just to define where words begin and end. For Arabic, you deal with clitics and morphological affixes that attach to word roots. Each language family requires specialized knowledge and specialized code.

These pretokenization rules introduce several problems that compound over time. First, they create language-specific dependencies. A tokenizer designed for English often produces garbage output when fed Japanese text, because the English tokenizer's rules about spaces and punctuation do not apply. Maintaining a separate tokenization system for each of the world's major languages is tedious and creates divergence in behavior that makes cross-lingual analysis unreliable. Second, language-specific rules make the tokenizer brittle: a word that was accidentally run together without a space becomes a single opaque unit that the subword algorithm cannot decompose. Third, pretokenization rules bake in assumptions about what a "word" is, assumptions that reflect the researchers' linguistic intuitions rather than the patterns a neural network needs to learn.

The deeper issue is conceptual. When you pretokenize, you commit to a particular segmentation of the input before training has even started. You are making a hard decision about word boundaries based on surface rules, then training the subword algorithm only within those boundaries. If your pretokenization is wrong or inconsistent, those errors propagate into every downstream model you train. Every model trained on that tokenizer inherits those errors and the idiosyncrasies of your pretokenization choices become baked into the representations that the model learns.

There is also a linguistic argument against pretokenization. Languages differ in their scripts and in how morphology works. English is relatively analytic: most meaning is carried by separate words ("walk", "walked", "walking"). Turkish is highly agglutinative: a single word can carry meaning that requires an entire English sentence to express. Finnish, Hungarian, and many other languages fall in between. Pretokenization rules designed for analytic languages do not transfer gracefully to agglutinative ones, because the concept of a "word" is fundamentally different.

SentencePiece eliminates the pretokenization step entirely. It feeds raw text directly to the subword algorithm, with no preprocessing except Unicode normalization. This means the algorithm can discover its own notion of meaningful units from data, including units that span what we might traditionally call word boundaries. In agglutinative languages, the algorithm naturally learns that word stems and common suffixes form useful token boundaries. In character-based languages like Chinese, it learns character-level and bigram-level units without needing an external word segmenter.