GPT-3 › Tokenization and Its Consequences for GPT-3

GPT-3 uses the same byte-pair encoding (BPE) tokenizer as GPT-2, with a vocabulary of 50,257 tokens. This tokenizer was trained on a corpus similar to WebText and represents a reasonable tradeoff between vocabulary size and coverage. Understanding the tokenizer's properties is important for understanding both GPT-3's capabilities and some of its failure modes.

BPE tokenization splits text into a sequence of subword units. Common words like "the", "is", and "language" are typically single tokens. Less common words are split into component subwords: "tokenization" might become ["token", "ization"]. Rare words or proper nouns may be split into individual characters or byte representations. The exact splits depend on the frequency statistics of the training corpus.

This tokenization has several consequences for GPT-3's behavior. First, the model processes text at the token level, not the character level. This means tasks that require character-level reasoning, such as counting letters, reversing strings, or identifying if a word is a palindrome, are difficult because GPT-3 does not have direct access to character-level information. A word like "level" is represented as a single token; to reverse it, the model would need to reason about the characters that compose that token, which is not a natural operation for a model trained on token sequences.

Second, the tokenization introduces a specific granularity for arithmetic. Numbers are tokenized in ways that do not correspond to their place-value structure. The number "1234" might be tokenized as a single token, or as "123" and "4", or as "12" and "34", depending on what subword units were learned. This inconsistent granularity helps explain GPT-3's arithmetic performance: the model can learn arithmetic operations on token sequences, but the fact that the same number can be tokenized differently in different contexts makes reliable computation difficult. A model with a character-level tokenizer would have more consistent access to the digit structure needed for reliable arithmetic.

Third, the tokenizer handles different languages with varying efficiency. Languages that appear frequently in the training corpus have more specialized tokens, requiring fewer tokens to represent the same amount of text. Languages that appear rarely are encoded more inefficiently, requiring more tokens per word. This efficiency difference translates to a context length disadvantage: GPT-3 can fit more English text in its 2,048 token context than equivalent text in a low-resource language, which disadvantages non-English speakers when using the model.

These tokenization effects are not unique to GPT-3, they affect all BPE-based models, but GPT-3's widespread deployment made them practically significant. Understanding that "the model struggles with character-level tasks" is not a vague capability limitation but a specific consequence of how text is represented provides a much clearer basis for predicting where the model will fail and for designing prompts that work around the limitation.