Word Tokenization › Impact on Downstream Tasks

Tokenization choices ripple through the entire NLP pipeline in ways that are sometimes dramatic and sometimes subtle. The tokens you produce become the atoms of every subsequent computation, and small changes to tokenization can have surprisingly large effects.

Vocabulary size is the most direct consequence. Splitting contractions increases vocabulary slightly but improves coverage of negation patterns. Keeping compounds together reduces vocabulary but may hurt generalization to new compound forms. The relationship between vocabulary size and model performance is not simple: too-small vocabularies mean models can't distinguish fine-grained meaning differences, but too-large vocabularies mean rare words get poor embeddings.

Sequence length affects model training time and memory usage. More tokens mean longer sequences. For Transformer models, which have quadratic attention costs, doubling the sequence length quadruples the computation per sequence. Tokenization choices that increase average token count by even 20% can have significant effects on training efficiency.

Semantic alignment matters because tokens should ideally correspond to meaningful units. Poor tokenization can obscure semantic relationships. If "undo" and "do" are treated as completely independent tokens, a model must learn from scratch that they are related. If they share a subword representation (both contain "do"), the model gets this relationship for free.

Cross-lingual transfer is affected because inconsistent tokenization across languages makes multilingual models harder to train. If English text uses Penn Treebank tokenization but French text uses a different convention, the model sees different tokenization styles for what should be parallel content. Modern multilingual models use a shared subword vocabulary across all languages, which helps but requires careful design.