SentencePiece › Subword Regularization in Depth

Subword regularization is a training technique unique to the unigram model that has proven surprisingly powerful in practice. It deserves its own extended discussion because it illustrates a broader principle: using the ambiguity of a system as a feature rather than a bug.

Standard tokenization is deterministic. Given a piece of text and a trained tokenizer, there is exactly one output: the most probable (or rule-derived) segmentation. During model training, every time the model sees the word "running", it sees the same token sequence. The model learns to associate that exact token sequence with the word's meaning.

Subword regularization changes this by sampling from the distribution of possible segmentations at training time. For "running", the most probable unigram segmentation might be ["▁running"] if the word appears frequently enough in the vocabulary, or ["▁run", "ning"] if not. With subword regularization and a sampling temperature $\alpha$, the model sometimes sees ["▁running"], sometimes ["▁run", "n", "ing"], sometimes ["▁r", "unning"], and so on, each with probability proportional to the segmentation's probability raised to the power $\alpha$.

This stochastic segmentation acts as a form of data augmentation. You can think of it as adding noise to the input representation: instead of always giving the model the same token sequence for a word, you give it a variety of plausible decompositions. The model must learn to produce the correct output regardless of how the input happens to be segmented, which forces it to learn that meaning is compositional across token boundaries.

The practical effect is most visible in three scenarios. First, for morphologically rich languages, regularization helps the model learn shared representations for morphologically related words even when they tokenize differently. Second, for translation, regularization reduces the brittleness of alignment: the model does not rely on both source and target text being segmented in exactly the one way. Third, for out-of-vocabulary words at inference time, a model trained with regularization handles unusual segmentations better because it has been trained to handle segmentation variability.

The sampling temperature $\alpha$ controls the strength of regularization. When $\alpha = 1$, you sample proportionally to the segmentation probabilities. When $\alpha \to 0$, you always use the most probable segmentation (equivalent to standard deterministic tokenization). When $\alpha \to \infty$, you sample uniformly from all segmentations. In practice, values around 0.1 to 0.5 tend to work well.