The Vocabulary Problem › Vocabulary Size Explosion
Language AI HandbookAll chapters The Vocabulary Problem › Vocabulary Size ExplosionOne section · Sections in this chapter3 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Managing the OOV problem requires making a difficult choice about vocabulary size. This choice sits at the heart of word-level system design, and every option comes with painful tradeoffs. Understanding these tradeoffs precisely is essential for appreciating why subword methods were such a breakthrough.
Think of vocabulary size as a dial with no good setting. Turn it down and you have a fast, memory-efficient system that understands very little of what users write. Turn it up and you have broader coverage at the cost of massive memory consumption, slow computation, and paradoxically worse embeddings for the rare words you're trying to include. The tradeoff isn't just technical: it reflects a deeper tension between the finite nature of computational resources and the infinite productivity of language.
The Coverage-Size Tradeoff
How large should your vocabulary be? This question reveals a fundamental tension in NLP system design that practitioners grapple with every time they train a new model.
A small vocabulary is computationally efficient. The embedding matrix, output projection layer, and any word-based operations all scale linearly with vocabulary size. A vocabulary of 10,000 words means 10,000 embeddings to store and 10,000 output classes for word prediction. Model parameters are proportional to vocabulary size: halving the vocabulary roughly halves the parameter count for embedding-heavy architectures. Training is faster, inference is faster, and memory consumption is manageable.
But a small vocabulary means a high OOV rate. Users will constantly encounter the [UNK] token, degrading model performance across every task. Text from specialized domains, from different time periods, or from users with non-standard spelling will be heavily affected. The model becomes brittle: it performs well on text similar to its training data and fails unpredictably on anything else.
A large vocabulary reduces OOV rates but introduces its own set of serious problems:
- Memory explosion: Each word needs an embedding vector. With 300-dimensional float32 embeddings, each word requires 1,200 bytes. At 1 million words, the embedding matrix alone consumes 1.2 GB. At 10 million words, it consumes 12 GB, often exceeding the memory of a single GPU and making distributed training a necessity.
- Sparse gradients: Rare words appear infrequently during training, so their embeddings receive few gradient updates and remain poorly learned. A word appearing 3 times in a billion-token corpus has an embedding that's essentially random noise, just as useless as
[UNK]but consuming memory. - Computational cost: The final layer of a language model computes a softmax distribution over the entire vocabulary to predict the next word. This operation scales linearly with vocabulary size. Computing softmax over 1 million classes requires a million floating-point exponentials and divisions per prediction step, which is prohibitively expensive during both training and inference without special approximations.
- Diminishing returns: Each word added to the vocabulary covers a progressively less frequent token. The 50,001st word in your vocabulary might appear once per 10 million tokens, contributing almost nothing to practical coverage while adding the same per-word cost as "the."
Let's visualize how vocabulary size affects corpus coverage, using the power-law structure of Zipf's law to model coverage as a function of vocabulary size.
The curve reveals a sobering truth: achieving high coverage requires exponentially larger vocabularies, but the returns diminish sharply. Going from 75% to 90% coverage means multiplying your vocabulary by 10, from roughly 1,000 words to 10,000. Going from 90% to 96% requires multiplying by 5 again, from 10,000 to 50,000 words. Going from 96% to 98% requires doubling again to 100,000 words. Each additional percentage point of coverage is more expensive than the last. And 100% coverage is essentially impossible because language is infinitely productive, with new words entering use every day. The curve never reaches 100%; it asymptotically approaches it while the vocabulary grows without bound.
Real-World Vocabulary Statistics
Let's examine actual vocabulary sizes from popular NLP resources to understand the practical scale of this problem and the resulting memory costs.
# Vocabulary sizes from real NLP resources (approximate)
vocabulary_stats = {
"Basic English": 850,
"Common English words": 3_000,
"Average adult vocabulary": 30_000,
"Shakespeare's works": 31_534,
"Oxford English Dictionary": 171_476,
"Word2Vec Google News": 3_000_000,
"FastText English": 2_000_000,
"GloVe 840B Common Crawl": 2_200_000,
"Google Web 1T 5-gram": 13_588_391,
}
# Memory requirements (assuming 300-dim float32 embeddings)
embedding_dim = 300
bytes_per_float = 4
memory_requirements = {}
for name, size in vocabulary_stats.items():
memory_bytes = size * embedding_dim * bytes_per_float
memory_mb = memory_bytes / (1024 * 1024)
memory_requirements[name] = memory_mbVocabulary Sizes and Memory Requirements (300-dim embeddings) ======================================================================== Resource Vocab Size Memory ------------------------------------------------------------------------ Basic English 850 996.1 KB Common English words 3,000 3.4 MB Average adult vocabulary 30,000 34.3 MB Shakespeare's works 31,534 36.1 MB Oxford English Dictionary 171,476 196.2 MB Word2Vec Google News 3,000,000 3.4 GB FastText English 2,000,000 2.2 GB GloVe 840B Common Crawl 2,200,000 2.5 GB Google Web 1T 5-gram 13,588,391 15.2 GB
The memory requirements tell a stark story. Basic English's 850 words need less than a megabyte for embeddings, while large pre-trained word vectors like Word2Vec or GloVe contain millions of entries requiring multiple gigabytes. Each additional word in the vocabulary adds exactly bytes (1.2 KB) for its 300-dimensional float32 embedding vector. At 3 million words, the embedding matrix alone consumes 3.4 GB, before any model weights. The Google Web 1T corpus, with over 13 million unique word forms, would require more than 14 GB just for the embedding table. This creates a practical ceiling on vocabulary size and forces practitioners to choose between coverage and computational efficiency.
The numbers in this table represent real engineering constraints. Modern GPU memory tops out at 40 to 80 GB for the most expensive hardware. A 14 GB embedding table leaves very little room for the rest of the model, attention layers, feed-forward networks, and batch activations during training. In practice, most word-level models cap their vocabularies at 50,000 to 100,000 words precisely because larger vocabularies become difficult to fit on a single accelerator, and gradient updates for rare words in multi-GPU setups require expensive communication. Vocabulary size is both a coverage decision and an infrastructure constraint.