The Vocabulary Problem › Why Character-Level Tokenization Doesn't Fully Solve It

At first glance, character-level tokenization seems to resolve the vocabulary problem elegantly. If you tokenize at the character level, there are only a few hundred possible tokens for any given writing system, and OOV is literally impossible for any text written in a supported script. No matter what word a user types, no matter how novel or misspelled, it can be represented as a sequence of characters that are all in the vocabulary. The vocabulary explodes to millions of word forms with word-level tokenization; with character-level tokenization, you need only 26 letters for English, plus some punctuation. Problem solved?

Character-level tokenization does not eliminate the issue; it solves the OOV problem by shifting it to a different level: the model must now learn to compose characters into meaningful units. Characters themselves carry little semantic information. The letter "c" doesn't mean anything until it combines with other letters into words and morphemes. A model operating on characters must learn, from scratch and implicitly, that the sequence "c-a-t" represents the same concept as "c-a-t-s" plus a plural marker, and that "c-h-a-t" is a different concept entirely. All the linguistic structure that word-level models get for free by using words as the basic unit must be rediscovered from character sequences.

The problem is not just conceptual: it is also computational. A single word like "transformers" becomes 12 characters. A typical sentence of 15 words might contain 75 to 100 characters including spaces. Modern transformer models have a limited context window, measured in tokens, so longer token sequences mean less text fits in memory. A 512-token context window can hold roughly 500 words of text at word-level tokenization, but only 100 to 125 words at character-level tokenization. Reducing the effective context length by a factor of 4 to 5 is a severe degradation for tasks requiring long-range reasoning like summarization, document understanding, and multi-turn dialogue.

Empirically, character-level models require more parameters and training data than word-level models to achieve equivalent performance on most tasks. Studies have shown that character-level language models need roughly 5 to 10 times more training tokens to match the perplexity of word-level models on the same text domain, because each prediction step carries less information and the model must expend capacity learning to compose characters into meaningful patterns. The information is technically present in the character sequence, but it is harder to extract and requires more model capacity and training to do so.

Subword tokenization sits in the sweet spot between words and characters. Subwords are longer than characters, so each token carries more semantic content and sequences are shorter. They are shorter than words, so novel terms can be represented through composition without requiring vocabulary entries for every surface form. A vocabulary of 30,000 to 50,000 subword units can represent virtually any text without OOV, at a sequence length overhead of only 10 to 30% compared to word-level tokenization. The key insight is that subwords capture the natural units of language: morphemes, common prefixes and suffixes, high-frequency word fragments that recur across many contexts.