Byte Pair Encoding › Vocabulary Size: The Core Tradeoff

Vocabulary size is where practitioners most often need to make a principled choice. The tradeoff operates along two axes simultaneously: sequence length and parameter count. Getting this balance right matters for training efficiency, inference cost, and downstream model quality.

Sequence length decreases as vocabulary size grows. A vocabulary of 1,000 tokens might encode a word like "computational" as 7 pieces, while a vocabulary of 50,000 might represent it as a single token. Shorter sequences are cheaper to process in transformers because attention is quadratic in sequence length. Every token you eliminate from the average sequence reduces inference cost. For models that process long documents, this effect is particularly significant: a 10% reduction in average sequence length translates directly to a roughly 20% reduction in attention computation.

Embedding parameters increase with vocabulary size. Each token requires a row in the embedding matrix. For a model with hidden dimension 4,096, a vocabulary of 50,000 tokens requires a 50,000 $\times$ 4,096 matrix with about 200 million parameters, just for the input embeddings. Doubling the vocabulary doubles this cost. For rare tokens that appear only a handful of times during training, these parameters may be poorly learned, effectively wasting capacity.

The practical sweet spot for most monolingual English models is 30,000 to 50,000 tokens. GPT-2 and RoBERTa both use approximately 50,000. BERT uses 30,522 (through WordPiece, a BPE variant). Models targeting multiple languages or code often use larger vocabularies. GPT-4's tokenizer uses over 100,000 tokens to accommodate the linguistic diversity of its training data.

The trend toward larger vocabularies in more recent models is not accidental. As models scale up and are deployed more broadly, the cost of a large embedding matrix becomes relatively smaller compared to the attention and feedforward layers, while the benefit of fewer tokens per sequence becomes relatively larger. LLaMA 3's jump to 128,256 tokens reflects both the model's multilingual ambitions and the practical observation that at billion-parameter scale, the embedding table is a small fraction of total parameters.