SentencePiece › A Worked Example

Let's trace through exactly how SentencePiece processes a concrete example, using BPE mode. We'll use the sentence "natural language processing" and watch the algorithm build its vocabulary from scratch. Following a complete example numerically is the best way to make the abstract algorithm concrete before encountering it in code.

The first transformation replaces spaces with ▁. SentencePiece also prepends a ▁ before the very first word, not just at internal space positions:

This is the only preprocessing step. There is no lowercasing, no punctuation normalization, no stemming. The raw Unicode string (minus the space-to-▁ substitution) feeds directly into the algorithm. Every character, including digits, punctuation, and characters from non-Latin scripts, is preserved exactly as-is.

BPE starts with a vocabulary containing every unique character (or Unicode code point) in the ▁-encoded corpus. For our sentence, the initial vocabulary contains all the unique characters:

The full text is now represented as 29 individual character tokens, one for every character position. Every character is a separate unit; there is no notion of "word" or "morpheme" yet. The algorithm is starting from the most granular possible representation and will build structure from the bottom up.

Notice that ▁ is in the initial vocabulary as a regular character. It has no special status beyond the fact that it happens to encode word boundary information. The BPE algorithm treats it exactly the same as any letter.

The algorithm counts every pair of adjacent characters in the corpus and merges the most frequent one. With a tiny single-sentence corpus, the counts might look like the table below. In a real training corpus with millions of sentences, the frequencies would be much larger but the principle is identical.

The pair (a, l) wins with count 3. We merge it into a new token "al" and replace all occurrences in the corpus:

Notice the corpus now has 27 tokens instead of 29: two pairs of (a, l) were replaced, each saving one token. This is the compression benefit made concrete. The algorithm continues, perhaps next merging (a, n) → "an", then (n, g) → "ng", and eventually (i, n, g) → "ing" by first merging "in" then "ing". After enough merges, common morphological units emerge naturally: "-ing", "-tion", "▁un", and so on.

The key insight from tracing this example is that the algorithm never explicitly looks for morphological units. It just looks for frequent pairs. Morphological units emerge naturally because morphemes by definition are recurrent patterns. The suffix "-ing" appears in "processing", "understanding", "learning", and countless other words. That frequency makes it a natural merge candidate. The algorithm discovers morphological structure as a consequence of optimizing for frequency, not because it was designed to find morphemes.

Once training is done, tokenizing new text means applying the learned merge operations in order. The text "▁natural▁language" starts as individual characters. Then we apply merge 1 (a, l) → "al", which converts "a, l" pairs to "al". We apply merge 2, then merge 3, and so on through the full list of merges in the order they were learned.

The same merge sequence learned during training produces a deterministic result on any input text. This ordered application of merges is what makes BPE fast at inference: it is just a sequence of string substitutions that can be implemented as a simple loop or compiled into efficient pattern-matching code. No probabilities need to be computed, no dynamic programming is needed. You just apply the rules in order.

One important detail: if new text contains a character that was not in the training corpus (and therefore not in the initial vocabulary), BPE cannot represent it. This is the unknown character problem. The unigram model's byte_fallback feature, which we will see in the code section, solves this by falling back to UTF-8 byte tokens for unknown characters.