SentencePiece › Code Implementation
Language AI HandbookAll chapters SentencePiece › Code ImplementationOne section · Sections in this chapter6 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's implement SentencePiece from scratch, then use the production library. Building it yourself reveals exactly what is happening inside the library; using the production library shows how to configure it for real applications.
Installation
First, let's install the required packages:
import subprocess
subprocess.run(["uv", "pip", "install", "sentencepiece"], capture_output=True)Implementing BPE from Scratch
Our SimpleSentencePiece class implements the full BPE pipeline: preprocessing with ▁ markers, character-level initialization, iterative merging, and inference-time tokenization. Building this from scratch forces you to confront every design decision that the production library handles automatically.
import collections
from typing import Dict, List, Tuple
class SimpleSentencePiece:
"""Minimal BPE tokenizer with SentencePiece-style whitespace handling."""
def __init__(self, vocab_size: int = 100):
self.vocab_size = vocab_size
self.vocab: Dict[str, int] = {}
self.merges: List[Tuple[Tuple[str, str], str]] = []
def _preprocess(self, text: str) -> str:
"""Replace spaces with ▁ prefix to encode word boundaries."""
return "▁" + text.replace(" ", "▁")
def train(self, corpus: List[str]) -> None:
"""Train BPE on the corpus and learn merge operations."""
# Preprocess all texts
processed = [self._preprocess(text) for text in corpus]
text_list = list(" ".join(processed))
# Initialize vocabulary with unique characters
chars = sorted(set(text_list))
self.vocab = {ch: i for i, ch in enumerate(chars)}
token_id = len(self.vocab)
while len(self.vocab) < self.vocab_size:
# Count frequencies of all adjacent pairs
stats: Dict[Tuple[str, str], int] = collections.defaultdict(int)
for i in range(len(text_list) - 1):
stats[(text_list[i], text_list[i + 1])] += 1
if not stats:
break
# Select and merge the most frequent pair
best_pair = max(stats, key=stats.get)
new_token = "".join(best_pair)
self.vocab[new_token] = token_id
self.merges.append((best_pair, new_token))
token_id += 1
# Apply merge to the running text representation
i = 0
while i < len(text_list) - 1:
if (text_list[i], text_list[i + 1]) == best_pair:
text_list[i : i + 2] = [new_token]
else:
i += 1
def tokenize(self, text: str) -> List[str]:
"""Tokenize text by preprocessing then applying learned merges."""
tokens = list(self._preprocess(text))
for pair, new_token in self.merges:
i = 0
while i < len(tokens) - 1:
if (tokens[i], tokens[i + 1]) == pair:
tokens[i : i + 2] = [new_token]
else:
i += 1
return tokens
def encode(self, text: str) -> List[int]:
"""Convert text to token IDs."""
tokens = self.tokenize(text)
return [self.vocab.get(t, self.vocab.get("▁", 0)) for t in tokens]
def decode(self, ids: List[int]) -> str:
"""Convert token IDs back to text."""
id_to_token = {v: k for k, v in self.vocab.items()}
tokens = [id_to_token.get(i, "") for i in ids]
return "".join(tokens).replace("▁", " ").strip()The _preprocess method does the single key transformation: it prepends ▁ and replaces every space with ▁. The train method initializes the vocabulary from individual characters, then iteratively finds and merges the most frequent pair. The tokenize method applies those learned merges in order to new text. The decode method simply reverses the ▁ substitution to recover the original string.
Training on a Small Corpus
Let's train on a small NLP-themed corpus and observe what patterns emerge. Even with a small corpus, the algorithm discovers structure:
corpus = [
"natural language processing",
"machine learning models",
"natural language understanding",
"deep learning algorithms",
"processing natural language",
"understanding machine learning",
"deep neural networks",
"learning language models",
"language model training",
"neural network architecture",
]
sp = SimpleSentencePiece(vocab_size=60)
sp.train(corpus)
test_text = "natural language"
tokens = sp.tokenize(test_text)
token_ids = sp.encode(test_text)
decoded = sp.decode(token_ids)
initial_chars = len(set("".join([sp._preprocess(t) for t in corpus])))
num_merges = len(sp.merges)
compression_ratio = len(sp._preprocess(test_text)) / len(tokens)Input text: 'natural language' Preprocessed: '▁natural▁language' Tokens: ['▁natural', '▁language'] Token IDs: [44, 33] Decoded: 'natural language' Initial vocabulary: 19 unique characters Final vocabulary: 60 tokens Merges learned: 40 Compression ratio: 8.50x
The decoded output matches the original input exactly, confirming that the ▁ encoding is perfectly invertible. The compression ratio shows how many fewer tokens we need compared to individual characters. Even on this tiny corpus, the algorithm has compressed the representation by merging common patterns. Notice that the ▁ character appeared in the initial vocabulary as a regular character, and some of the learned merges likely include it as part of the pattern.
Examining Learned Merges
The merge sequence reveals exactly what patterns the algorithm discovered. Early merges capture the most frequent character pairs; later merges combine previously merged units into longer subwords. Let's look at the progression:
First 8 learned merges (most frequent character pairs): 1. 'n' + 'g' -> 'ng' 2. '▁' + 'l' -> '▁l' 3. 'i' + 'ng' -> 'ing' 4. '▁' + 'n' -> '▁n' 5. 'd' + 'e' -> 'de' 6. 'u' + 'r' -> 'ur' 7. 'a' + 'l' -> 'al' 8. 'ur' + 'al' -> 'ural' Last 5 learned merges (longer subwords): 36. 'pro' + 'c' -> 'proc' 37. 'proc' + 'e' -> 'proce' 38. 'proce' + 's' -> 'proces' 39. 'proces' + 's' -> 'process' 40. 'process' + 'ing' -> 'processing'
The progression from single characters to morphological units illustrates how BPE discovers structure hierarchically. Early merges address the most common character bigrams, which in English-like text are often consonant-vowel pairs. Later merges build on these, creating longer tokens that correspond to common morphological units. The ▁ character participates in merges just like any other character, which is why many early tokens include the word-boundary prefix. A token like "▁la" in the vocabulary means "la" at the start of a word, a pattern that appears in "language", "language", and "learning".
Visualizing BPE Training Dynamics
To understand how vocabulary growth relates to compression efficiency, let's instrument the training loop and track both metrics across all merge iterations:
The dual-axis view makes the trade-off visible. Each merge adds exactly one token to the vocabulary (the blue line rises in unit steps). The corpus compression (token count, orange) falls fastest at the beginning because those first merges target the most frequent character pairs, which appear throughout the corpus. As training continues, each additional merge produces smaller compression gains because it is addressing rarer patterns that appear in fewer positions. This diminishing return is fundamental to BPE: the vocabulary grows linearly, but the useful work each new token does decreases.
Vocabulary Size and Compression Trade-offs
The choice of vocabulary size is one of the most consequential decisions when configuring SentencePiece. A vocabulary that is too small forces the model to represent many words as long sequences of short tokens, which makes sequence lengths longer and harder for a transformer's attention mechanism to handle. A vocabulary that is too large creates a massive embedding table that consumes memory and makes the softmax over vocabulary at the output of a language model expensive. Let's empirically measure how compression efficiency changes as vocabulary grows:
The diminishing returns are clear. Moving from 20 to 40 tokens produces a significant compression gain. Moving from 60 to 75 produces much less. In production systems like BERT (30,522 tokens) or T5 (32,100 tokens), the vocabulary is large enough that most common English words receive their own token, but still small enough that the embedding table remains manageable. The typical range of 8,000 to 32,000 tokens represents the empirically discovered sweet spot where compression is good but the vocabulary remains tractable.
Using the Production SentencePiece Library
Our from-scratch implementation teaches the mechanics. For production use, the official sentencepiece library provides a much faster, more configurable implementation that handles edge cases properly. The library is written in C++ with Python bindings and can tokenize millions of tokens per second on a single CPU core.
First, we create a training corpus file and train a model:
import os
import tempfile
import sentencepiece as spm
# Write corpus to temporary file
extended_corpus = corpus + [
"tokenization is fundamental",
"multilingual model training",
"byte pair encoding algorithm",
"unigram language model tokenizer",
"subword regularization technique",
"vocabulary size selection",
]
with tempfile.NamedTemporaryFile(mode="w", suffix=".txt", delete=False) as f:
for line in extended_corpus:
f.write(line + "\n")
corpus_file = f.name
# Train with BPE
with tempfile.TemporaryDirectory() as tmpdir:
model_prefix = os.path.join(tmpdir, "sp_model")
spm.SentencePieceTrainer.train(
input=corpus_file,
model_prefix=model_prefix,
vocab_size=200,
model_type="bpe",
character_coverage=1.0,
pad_id=0,
unk_id=1,
bos_id=2,
eos_id=3,
)
# Load and save the model bytes for later use
with open(model_prefix + ".model", "rb") as f:
model_bytes = f.read()
os.unlink(corpus_file)
# Load model from bytes
sp_prod = spm.SentencePieceProcessor()
sp_prod.load_from_serialized_proto(model_bytes)Notice that training produces a binary .model file that encodes the complete tokenizer state: the vocabulary, the merge rules (for BPE) or the token probabilities (for unigram), and all configuration parameters. This file is everything you need to reproduce the exact same tokenization in any environment. You can load it in Python, C++, or any language with a SentencePiece binding.
# Tokenize several texts
test_cases = [
"natural language processing",
"deep neural network",
"multilingual tokenization",
"unknown words like tokenizationing",
]
tokenization_results = []
for text in test_cases:
pieces = sp_prod.encode_as_pieces(text)
ids = sp_prod.encode_as_ids(text)
decoded = sp_prod.decode_pieces(pieces)
tokenization_results.append(
{
"text": text,
"pieces": pieces,
"ids": ids,
"decoded": decoded,
"n_tokens": len(pieces),
}
)Production SentencePiece tokenization results: Input: 'natural language processing' Pieces: ['▁natural', '▁language', '▁processing'] Tokens: 3 Decoded: 'natural language processing' Input: 'deep neural network' Pieces: ['▁deep', '▁neural', '▁network'] Tokens: 3 Decoded: 'deep neural network' Input: 'multilingual tokenization' Pieces: ['▁multilingual', '▁tokenization'] Tokens: 2 Decoded: 'multilingual tokenization' Input: 'unknown words like tokenizationing' Pieces: ['▁un', 'k', 'n', 'o', 'w', 'n', '▁', 'word', 's', '▁l', 'i', 'k', 'e', '▁token', 'izat', 'io', 'ning'] Tokens: 17 Decoded: 'unknown words like tokenizationing'
The production library handles the ▁ prefix, BPE or unigram selection, special tokens (PAD, UNK, BOS, EOS), and multilingual text all transparently. Notice how "multilingual tokenization" is handled even though those exact words were not in the training corpus: the algorithm decomposes them into known subword units. This graceful degradation on unseen words is exactly why subword tokenization replaced word-level tokenization. Even a word that was never seen during training can be decomposed into known pieces.
Multilingual Tokenization
One of SentencePiece's most important properties is handling multiple scripts in a single vocabulary. A single vocabulary can cover dozens of languages simultaneously because SentencePiece works at the level of Unicode characters, not language-specific word lists. Let's see how it handles text from different languages by training on a multilingual corpus:
multilingual_corpus = [
# English
"natural language processing",
"machine learning models",
"deep learning algorithms",
# German
"natürliche Sprachverarbeitung",
"maschinelles Lernen",
"tiefes Lernen",
# Spanish
"procesamiento de lenguaje natural",
"aprendizaje automático",
"redes neuronales",
# French
"traitement automatique du langage",
"apprentissage automatique",
"réseaux de neurones",
]
with tempfile.NamedTemporaryFile(
mode="w", suffix=".txt", delete=False, encoding="utf-8"
) as f:
for line in multilingual_corpus:
f.write(line + "\n")
multi_corpus_file = f.name
with tempfile.TemporaryDirectory() as tmpdir:
model_prefix = os.path.join(tmpdir, "sp_multi")
spm.SentencePieceTrainer.train(
input=multi_corpus_file,
model_prefix=model_prefix,
vocab_size=300,
model_type="unigram",
character_coverage=1.0,
byte_fallback=True,
)
with open(model_prefix + ".model", "rb") as f:
multi_model_bytes = f.read()
os.unlink(multi_corpus_file)
sp_multi = spm.SentencePieceProcessor()
sp_multi.load_from_serialized_proto(multi_model_bytes)
# Test tokenization across languages
multi_test = [
("EN", "language model"),
("DE", "Sprachmodell"),
("ES", "modelo de lenguaje"),
("FR", "modèle de langage"),
]
multi_results = []
for lang, text in multi_test:
pieces = sp_multi.encode_as_pieces(text)
multi_results.append(
{"lang": lang, "text": text, "pieces": pieces, "n": len(pieces)}
)Multilingual tokenization (unigram model, character_coverage=1.0):
[EN] 'language model'
['▁', 'l', 'a', 'n', 'g', 'u', 'age', '▁', 'm', 'o', 'd', 'e', 'l']
(13 tokens)
[DE] 'Sprachmodell'
['▁', 'S', 'p', 'r', 'a', 'c', 'h', 'm', 'o', 'd', 'e', 'l', 'l']
(13 tokens)
[ES] 'modelo de lenguaje'
['▁', 'm', 'o', 'd', 'e', 'l', 'o', '▁de', '▁', 'l', 'e', 'n', 'g', 'u', 'a', 'j', 'e']
(17 tokens)
[FR] 'modèle de langage'
['▁', 'm', 'o', 'd', '<0xC3>', '<0xA8>', 'l', 'e', '▁de', '▁', 'l', 'a', 'n', 'g', 'age']
(15 tokens)A single vocabulary handles text from multiple languages simultaneously. The shared subword vocabulary allows multilingual models to transfer knowledge between languages: if the model learns that the English prefix "▁un-" signals negation, and it also sees similar negative prefixes in other languages, it can learn shared representations that help with cross-lingual transfer. This property is one of the reasons multilingual models like mBERT and XLM-RoBERTa can perform zero-shot cross-lingual transfer, applying knowledge learned in one language to tasks in a different language.
The character_coverage=1.0 parameter ensures every character in the training data is covered. The byte_fallback=True parameter ensures that characters not seen during training are encoded as their UTF-8 byte sequences rather than mapping to an unknown token. These two parameters together ensure consistent behavior across scripts.
Key Parameters
When training SentencePiece in production, several parameters have the largest impact on quality. Understanding what each parameter controls, and why you might want to change it, is essential for building models that work well on your specific data.
The key parameters for production use are:
- vocab_size: The total number of subword tokens. Production models typically use 8,000 to 32,000 tokens. Larger values improve compression and reduce token count per sentence; smaller values reduce memory usage. The 32,100 used by T5 and the 32,000 used by LLaMA are common choices that reflect the empirical finding that this range provides good compression without an unwieldy embedding table.
- model_type: Either
'bpe'for greedy frequency-based merging or'unigram'for the probabilistic EM approach. BPE is slightly faster at inference; unigram supports subword regularization during training. - character_coverage: The fraction of characters in the training data to include. Set to 1.0 for multilingual models to ensure full character coverage. For language-specific models, 0.9995 is sometimes used to exclude the rarest characters that appear only in noise or foreign text.
- byte_fallback: When True, any character not covered by the vocabulary is encoded as UTF-8 byte tokens. needed for less brittleness in multilingual settings. This ensures the tokenizer never produces unknown tokens.
- pad_id, unk_id, bos_id, eos_id: IDs for special tokens. Setting these explicitly ensures consistent IDs across model versions and training runs. Setting to -1 disables that special token.