Byte Pair Encoding › BPE Hyperparameters

Beyond vocabulary size, a few other decisions shape how a BPE tokenizer behaves. These choices are often invisible when you use a pre-trained tokenizer, but they have significant effects on tokenization quality and downstream model performance. Understanding them is important both for evaluating existing tokenizers and for training new ones.

Production BPE tokenizers do not operate on raw Unicode streams. They first apply a pre-tokenization step that splits text into rough word-level chunks, then apply BPE within each chunk. GPT-2 uses a regex that splits on whitespace and punctuation while preserving contractions and common abbreviations. Pre-tokenization prevents BPE merges from bleeding across word boundaries, which would produce tokens like "the" + "cat" merged together, spanning two words.

Pre-tokenization decisions have cascading effects. The GPT-2 regex treats leading spaces as part of the word, so "cat" and " cat" (with a preceding space) produce different tokens. This means the token for "cat" at the start of a sentence is different from the token for "cat" in the middle of a sentence. While this seems like a quirk, it provides the model with information about word position that it might not otherwise have. However, it also means that prompts with and without a leading space can produce different token sequences, which has occasionally caused unexpected behavior in applications.

Think of pre-tokenization as drawing hard boundaries that BPE merges cannot cross. Without these boundaries, BPE might decide that merging the final "e" of "the" with the initial "c" of "cat" is beneficial, because "e c" is a common sequence in English. That merge would be semantically useless and would clutter the vocabulary with cross-word artifacts. Pre-tokenization prevents this by treating word boundaries as inviolable.

A significant design choice made by GPT-2 was to operate on bytes rather than Unicode characters. Instead of initializing the vocabulary with potentially tens of thousands of possible Unicode code points, byte-level BPE starts with exactly 256 base tokens (one per byte value). This guarantees that any sequence of bytes, regardless of encoding, can be represented without unknown tokens.

Byte-level BPE has become the standard for large language models precisely because it is universal: you can feed any text, including emojis, mathematical symbols, CJK characters, or arbitrary binary content, and the tokenizer will always produce a valid encoding. The tiktoken library used by OpenAI's models implements byte-level BPE. The tradeoff is that non-ASCII characters may require multiple bytes, and therefore multiple tokens, to represent. A single Chinese character encoded in UTF-8 typically uses three bytes, so it initially costs three tokens. However, common characters appear frequently enough that BPE learns to merge those bytes into single tokens, recovering efficiency for high-frequency characters.

The key insight behind byte-level BPE is that it separates the question of "what can be represented" from "what is represented efficiently". All byte sequences can always be represented (using individual byte tokens as a fallback), while frequently occurring patterns are represented compactly through learned merges. This is a more principled solution than Unicode-character BPE, which can encounter truly unknown tokens for rare Unicode code points that never appeared in the training corpus.

The merge rules learned by BPE reflect the language of the training text. A tokenizer trained on English news articles will tokenize source code inefficiently, spending many tokens on common keywords like "function" that would merit their own token in a code-focused vocabulary. A multilingual tokenizer trained on 100 languages must allocate vocabulary space across all of them, which typically means each language gets fewer dedicated tokens than a monolingual tokenizer would assign it.

In practice, this means that using a pre-trained tokenizer for a domain very different from its training data can noticeably increase inference cost. A model tokenizer trained on general English may require six tokens for a common SQL keyword that would be a single token in a code-specialized tokenizer. This inefficiency compounds when processing long documents: a 2,000-word legal contract might require 50% more tokens when processed by a general tokenizer versus a legal-domain tokenizer, directly increasing inference cost by 50%.

This mismatch problem has motivated a growing practice of training domain-specific tokenizers for specialized models. Code models like Codex use tokenizers trained predominantly on code repositories, which allocate vocabulary space to common code patterns like indentation sequences, common variable names, and language keywords. Medical models may train tokenizers on clinical notes and scientific papers to ensure that medical terminology gets efficient representation.