Tokenizer Training › Vocabulary Size Selection
Language AI HandbookAll chapters Tokenizer Training › Vocabulary Size SelectionOne section · Sections in this chapter3 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Vocabulary size is the most impactful hyperparameter in tokenizer training. It controls the tradeoff between sequence length and vocabulary coverage. Getting this right matters because the vocabulary size is one of the few hyperparameters you cannot easily change after training begins. Once you have trained a model on a tokenizer with 32,000 vocabulary entries, you are committed to that vocabulary for the model's lifetime.
Think of vocabulary size as a budget. You have a fixed number of slots to allocate to meaningful character sequences. Every slot you allocate to a common suffix like "ing" or "tion" is a slot you are not allocating to a domain-specific term like "transformer" or "embedding". Every slot you allocate to a rare proper noun is a slot not allocated to a productive morpheme. The vocabulary size determines how many of these allocations you can make, and the training algorithm determines which allocations are made within that budget.
The total number of unique tokens in the tokenizer's vocabulary, including special tokens. Larger vocabularies produce shorter sequences but require more embedding parameters. Typical values range from 30,000 to 100,000 for general-purpose models. The vocabulary size directly determines the size of the embedding table: a model with a 50,000-token vocabulary and 768-dimensional embeddings has 38.4 million parameters just in the embedding layer.
The Vocabulary Size Tradeoff
Consider what happens at the extremes. Both extremes cause problems, and understanding why helps you reason about the middle ground.
Very small vocabulary (e.g., 256 bytes): Every word is split into many tokens, creating long sequences that are slow to process and hard for attention to span. A 256-byte vocabulary reduces every input to a sequence of individual bytes. The model must learn that bytes d, e, f together form the keyword def, and that d, e, f, space means the start of a function. This is learnable in principle but requires far more training data and model capacity than recognizing def as a single semantic unit.
Very large vocabulary (e.g., 1 million tokens): Most words are single tokens, but the embedding table becomes enormous and rare tokens have poor representations from insufficient training examples. A token that appears only three times in the training corpus will have an embedding that was updated only three times during training, which is nowhere near enough for the model to learn a reliable representation. Rare tokens end up with near-random embeddings that contribute noise rather than signal.
The sweet spot depends on your model size, training data, and target languages. Here is how vocabulary size affects tokenization in practice:
vocab_sizes = [100, 500, 1000, 5000, 10000]
tokenizers_by_size = {}
training_corpus = [
"Natural language processing enables computers to understand human language.",
"Machine learning models learn patterns from large datasets.",
"Deep learning uses neural networks with multiple hidden layers.",
"Transformers use attention mechanisms to model long-range dependencies.",
"The quick brown fox jumps over the lazy dog.",
"Recurrent networks process sequences one token at a time.",
"Convolutional layers extract local features from nearby inputs.",
"Pretraining on unlabeled data improves downstream performance.",
"Subword tokenization balances vocabulary coverage and sequence length.",
"Byte-pair encoding iteratively merges the most frequent character pairs.",
"WordPiece chooses merges that maximize the likelihood of the training data.",
"Unigram language models assign probabilities to subword segmentations.",
"Vocabulary size controls the tradeoff between sequence length and embedding cost.",
"Rare words are split into smaller subword units by the tokenizer.",
"Domain-specific tokenizers reduce fragmentation on specialized text.",
"Medical terminology often requires dedicated vocabulary for efficient encoding.",
"Legal documents contain long compound nouns that benefit from subword merges.",
"Code tokenizers preserve keywords, operators, and common identifiers intact.",
"Multilingual tokenizers share subword units across typologically diverse languages.",
"SentencePiece handles whitespace as part of the token rather than a separator.",
"Normalization converts text to a canonical form before subword splitting.",
"Lowercasing reduces vocabulary size by collapsing case variants.",
"Unicode normalization ensures consistent handling of accented characters.",
"Special tokens mark sentence boundaries and separate input segments.",
"The padding token allows batches of variable-length sequences to be processed together.",
] * 80
for size in vocab_sizes:
tokenizers_by_size[size] = train_tokenizer_on_corpus(
training_corpus, vocab_size=size
)
test_text = "Transformers process natural language efficiently."Test: 'Transformers process natural language efficiently.'
------------------------------------------------------------
Vocab Size | Tokens | Tokenization
------------------------------------------------------------
100 | 29 | T r an s for m ers pro ces s ...
500 | 12 | Tran sform ers process n at ural language effic ient ...
1000 | 9 | Transformers process n at ural language efficient ly .
5000 | 9 | Transformers process n at ural language efficient ly .
10000 | 9 | Transformers process n at ural language efficient ly .The results demonstrate a clear inverse relationship between vocabulary size and token count. With only 100 vocabulary slots, common words like "Transformers" get split into many character-level fragments. As vocabulary size increases to 5,000 and beyond, common words and morphemes become single tokens, dramatically reducing sequence length. The key insight is that this compression is not free: each additional vocabulary slot represents a distinct embedding vector the model must learn, plus a row in the output projection matrix for language modeling. You are trading memory and training cost for shorter, more semantically coherent sequences.
Fertility: A Practical Quality Metric
To reason quantitatively about vocabulary size choices, practitioners use a metric called fertility, defined as the average number of tokens per word. A tokenizer with fertility 1.0 turns every word into exactly one token. A tokenizer with fertility 3.0 splits every word into three pieces on average.
We compute fertility on a held-out sample of the target domain text:
A fertility above 1.5 on your target domain text is a warning sign that the tokenizer is fragmenting domain vocabulary excessively. You should either increase the vocabulary size or retrain on a more representative corpus. A fertility between 1.0 and 1.3 is generally healthy for single-language models. Multilingual models typically have higher fertility because the same vocabulary budget must cover multiple scripts and word formation patterns.
The fertility metric is also useful for comparing a new tokenizer against an existing baseline. If your custom tokenizer has fertility 1.2 on medical text while BERT's tokenizer has fertility 2.4 on the same text, you have quantified the advantage of domain-specific training.
Guidelines for Vocabulary Size Selection
There is no universal optimal vocabulary size, but production models converge on a common range that reflects empirical experimentation at scale. The following table shows vocabulary sizes for major production language models:
| Model | Vocabulary Size | Notes |
|---|---|---|
| GPT-2 | 50,257 | Byte-level BPE, English-focused |
| GPT-4 | 100,277 | Expanded for multilingual and code |
| BERT | 30,522 | WordPiece, English uncased |
| LLaMA | 32,000 | SentencePiece, efficient for inference |
| T5 | 32,128 | SentencePiece Unigram |
For domain-specific models, smaller vocabularies often work well because you do not need to cover the full breadth of general language:
- Legal or medical domains: 16,000-32,000. The domain vocabulary is specialized but limited in total size. Most of the vocabulary budget can be allocated to domain-specific terminology rather than general morphemes.
- Code models: 32,000-50,000. You need tokens for keywords, operators, and common identifiers. Python alone has roughly 35 keywords plus hundreds of standard library identifiers that appear frequently enough to merit vocabulary entries.
- Multilingual models: 100,000 or more. Multiple scripts, each with their own character inventory, consume vocabulary budget quickly. Chinese and Japanese each have thousands of characters that must be represented.
The rule of thumb is to start with 32,000 for single-language models and measure the fertility on a held-out sample of your target domain. If fertility is above 1.5, consider increasing vocabulary size. If your embedding table would exceed available GPU memory at your planned model size, reduce it. A useful formula to check: if your model has vocabulary entries and embedding dimensions, the embedding table has parameters. At vocabulary size 32,000 and embedding dimension 768, that is roughly 24.6 million parameters, already a substantial fraction of smaller models.