Word Tokenization › Word Tokenization
Language AI HandbookAll chapters Word Tokenization › Word TokenizationOne section · Sections in this chapter1 / 17 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Before a computer can understand language, it must break text into meaningful units. Word tokenization is the process of splitting text into individual words or tokens. It sounds trivial: just split on spaces, right? But human language has irregular boundaries and edge cases that trip up the naive approach. Contractions like "don't" and "we'll" blur word boundaries. Punctuation clings to words in complex ways. Abbreviations like "Dr." end with a period that is part of the word, not a sentence boundary. Some languages don't use spaces at all. This chapter explores why tokenization is harder than it looks, why the choices you make matter enormously, and how to do it well.
Think of tokenization as the act of drawing lines between words on a printed page. For clean, simple prose, the task seems mechanical. But the moment you encounter a hyphenated compound like "state-of-the-art", an email address like "user@example.com", or a mathematical expression like "$50.25", you realize there is no single right answer. Every line you draw reflects a decision about what counts as a meaningful unit, and those decisions have consequences for everything that follows.
Tokenization is the first step in nearly every NLP pipeline. Get it wrong, and errors cascade through everything downstream. A vocabulary that treats "waiting", "waiting.", and "waiting," as three separate words is bloated and inconsistent. A model that never sees "don't" because it only learned "don" and "t" will struggle with simple negation. A system that keeps "New York" as two separate tokens might fail to recognize it as a place name. Get tokenization right, and you've built a solid foundation for text analysis and search systems, as well as language models.
In this chapter, we will move from the simplest possible approach, splitting on whitespace, through progressively more sophisticated techniques. Along the way, we will handle punctuation and contractions, then address abbreviations and language-specific challenges. We will implement a rule-based tokenizer from scratch, compare against production-grade libraries like NLTK and spaCy, and examine how to evaluate tokenization quality. We will also take a brief look at how word tokenization connects to the subword methods used in modern neural NLP systems.
This chapter sits at the foundation of everything that follows in the book. The vocabulary you build from tokens becomes the input space for frequency analysis, the basis for term-frequency calculations, and eventually the foundation for word embeddings. Every subsequent technique assumes that text has been split into meaningful units. Understanding what "meaningful" means, and why the answer is not always obvious, is what this chapter is about.
Word tokenization has been a research problem since the early days of computational linguistics in the 1960s and 1970s, when researchers first attempted to build programs that could process natural language text. Early systems often assumed simple whitespace splitting was sufficient, a reasonable assumption for the clean typeset text they worked with. The problem became more acute in the 1980s and 1990s as NLP researchers began working with large corpora of real-world text. The Penn Treebank project (1989-1992) at the University of Pennsylvania produced the first widely adopted tokenization standard, annotating one million words of Wall Street Journal text with careful token boundaries. Their conventions, designed for English text, became the de facto standard for English NLP for two decades. The emergence of the World Wide Web in the 1990s introduced new challenges, including structured strings such as URLs and email addresses, along with hashtags and emoji that no rule set had anticipated. Today, modern neural NLP systems often sidestep word tokenization entirely by using subword methods that learn token boundaries from data, but understanding word tokenization remains essential for working with classical NLP tools, interpreting pre-neural research, and designing effective preprocessing pipelines.