SentencePiece › Whitespace as a Character: The ▁ Prefix
Language AI HandbookAll chapters SentencePiece › Whitespace as a Character: The ▁ PrefixOne section · Sections in this chapter3 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
If you feed raw text to a BPE or unigram algorithm without any whitespace handling, you lose all information about where words begin. The sequence "language model" would be indistinguishable from "languagemodel". The merge algorithm would treat the space like any other character, potentially creating tokens that straddle word boundaries in ways that make no linguistic sense and that would be impossible to decode unambiguously back to the original text.
SentencePiece handles this with a simple but powerful encoding. Before applying any subword algorithm, it replaces every whitespace character with a special Unicode symbol: ▁ (U+2581, the "Lower One Eighth Block"). This symbol is then treated as a regular character that can participate in merges and appear in token boundaries. The choice of U+2581 is deliberate: this block character is visually distinct from all standard punctuation and letters, it is unlikely to appear in natural text, and it survives most Unicode normalization schemes unchanged.
The key insight is that after this substitution, word boundary information is no longer encoded in empty space between characters but instead as a visible character that can participate in the algorithm on equal footing with letters and punctuation. When SentencePiece tokenizes "natural language processing", it first transforms the text to "▁natural▁language▁processing". Now each word carries its boundary marker as a prefix. The algorithms can learn that ▁ usually starts tokens, because it is followed by a consistent pattern of characters, and tokens without ▁ are continuations of a word.
When you decode a sequence of SentencePiece tokens back to text, you replace ▁ with a space. The token "▁natural" becomes " natural" (with a leading space), and adjacent tokens without ▁ are concatenated directly. This makes encoding and decoding perfectly invertible: you always get back exactly the original text. This invertibility is non-trivial. Many tokenization schemes are lossy in the sense that whitespace normalization or punctuation handling changes the input. SentencePiece guarantees that decode(encode(text)) == text for any input.
This encoding solves the word boundary problem without requiring language-specific rules. For English, ▁ appears where spaces were. For Chinese, where characters are written without spaces, ▁ may appear less frequently and the algorithm learns different patterns. For Japanese, which mixes character-based scripts with occasional spaces, the algorithm adapts accordingly. The same mechanism works for all three languages because the only assumption it makes is that some whitespace characters exist in the input and should be preserved in the output.
One practical consequence is that in SentencePiece output, you will often see tokens like "▁the", "▁language", or "▁un". The leading ▁ is not decoration; it carries semantic information. A token like "▁un" (meaning "un" at the start of a word) is different from "un" (meaning "un" in the middle of a word like "function"). This distinction matters because the same string in different positions has different distributional properties in text. The prefix "un-" as in "unhappy" or "unavailable" always appears at word beginnings. The substring "un" in "function" or "running" appears in the middle. Treating them as distinct tokens allows the model to learn these different distributional properties separately, which ultimately leads to better representations.
Think of the ▁ prefix as a kind of morphological annotation that the algorithm receives for free, without any linguistic knowledge. Instead of a linguist labeling which token positions are word-initial, the algorithm learns it directly from the encoding. This is another example of a recurring theme in modern NLP: replacing hand-crafted features with representations that can be learned from data.