Word Tokenization › Summary

Word tokenization breaks text into meaningful units for processing. While conceptually simple, the task is surprisingly complex because boundaries are affected by punctuation and language-specific conventions. Every tokenization decision is also a vocabulary decision, and every vocabulary decision shapes what patterns downstream systems can learn.

The journey through this chapter traces the evolution from naive whitespace splitting through progressively more sophisticated approaches. We started with text.split(), which works well for simple text but fails when punctuation or contractions complicate its output. We then extended it to protect abbreviations and split contractions, with each addition addressing a specific failure mode of the previous approach. We examined Penn Treebank conventions, which formalized English tokenization for the NLP community. We surveyed language-specific challenges that reveal how much English-centric assumptions are baked into common tokenizers. We implemented evaluation metrics that quantify how well a tokenizer matches human judgment.

Key takeaways:

• Whitespace tokenization is fast but leaves punctuation attached to words and is insufficient for most real NLP tasks
• Punctuation separation improves token quality but requires careful handling of abbreviations, decimal numbers, and domain-specific conventions
• Contractions can be kept together, split at the apostrophe, or expanded, and the best choice depends on the downstream task
• Penn Treebank conventions provide a widely-used standard for English tokenization that enables compatibility with pre-trained models
• Languages without spaces (Chinese, Japanese) require specialized segmenters that combine dictionary lookup and statistical modeling
• Compound words (German, Finnish) may need morphological decomposition for some applications
• Rule-based tokenizers are interpretable and deterministic but require extensive rule sets and degrade on unanticipated inputs
• Library tokenizers (NLTK, spaCy) handle common cases well and should be the default choice for production work
• Evaluation uses precision, recall, and F1 against gold standards; a good tokenizer needs high scores on both precision and recall
• Tokenization choices affect vocabulary size, sequence length, and semantic alignment, with ripple effects throughout the entire NLP pipeline

The right tokenization strategy depends on the application and the language and domain it targets. There's no universal best approach. Understanding the tradeoffs helps you make informed decisions for your specific application. As you move deeper into the book, you will see how the subword tokenization methods used by modern neural NLP systems address many of the limitations we've discussed here, while introducing their own tradeoffs that are worth understanding.